craft: A CRAFT Pipeline for Evaluating LLM-Generated Data

Status: v0.1.0, in active development. Feedback welcome.

craft operationalizes the C-R-A-F-T framework of Ko, Tai, and Webb Williams into 12 short R functions. It does not call LLM APIs itself: it takes the outputs you already have — labels, confidences, rationales — and walks them through five steps.

Step	Functions	What it does
Construct	`role()`	Documents how the LLM is being used (annotator, ML system, silicon participant) and which metric family applies.
Report	`reliab()`, `reliab_pairs()`, `valid()`, `dual()`	Reliability (Cohen’s κ, weighted κ, Fleiss’, Krippendorff’s α, ICC, percent agreement) paired with validity (precision, recall, F1, accuracy, balanced accuracy, MCC).
Assess	`stab()`	Cross-prompt and cross-model stability.
Field audit	`audit()`, `disagree()`, `tau_sens()`	Surfaces disagreements and low-confidence agreements; confidence-threshold sensitivity.
Translate	`dsl_fit()`, `dsl_cmp()`	Design-based supervised learning, so misclassification uncertainty propagates into inference.
Report	`report()`	Emits a reproducibility supplement capturing the model, version, prompt, metrics, thresholds, audit decisions, and DSL results.

Installation

# install.packages("remotes")
remotes::install_github("casstai/craft-r")

The dsl dependency (Egami et al.):

remotes::install_github("naoki-egami/dsl")

Quick start

library(craft)

# C: document the role-task
rt <- role("annotator", "classify climate stance",
           gold = TRUE, prompt_type = "few-shot")

# R: reliability and validity in one call
dual(ratings, gold = ratings$human, pred = ratings$gpt5,
     reliability_method = "kripp")

Documentation

Citation

If you use craft, please cite both the framework paper and the package:

Ko, Hyein, and Yuehong Cassandra Tai. “Can We Trust LLM-Generated Data? The CRAFT Framework for Measurement and Inference in Political Science.” Under review.

Tai, Yuehong Cassandra, and Hyein Ko. 2026. craft: A CRAFT Pipeline for Evaluating LLM-Generated Data. R package version 0.1.0. https://github.com/casstai/craft-r