Comparison
Exact-match cachingvsSemantic caching
Exact-match caching
the identical request comes in again and you return the stored answer, with no chance of it being a subtly different answer.
A plain cache on the normalised request. Only hits on identical inputs, but with zero wrong-answer risk; ideal for deterministic sub-steps (temperature 0 extraction, classification) inside larger pipelines.
Full entry →Semantic caching
the second person asks roughly the same question in different words and you serve them the stored answer without calling the model.
Caching keyed on meaning: embed the incoming query, and on high similarity to a cached query, return the cached answer without touching the model. The knob is the similarity threshold: too loose returns wrong answers, too tight never hits. Measure both rates on real traffic before trusting it.
Full entry →