Applied AI·Production patterns and cost
the second person asks roughly the same question in different words and you serve them the stored answer without calling the model.
Semantic caching
Draft summary, pending review
Caching keyed on meaning: embed the incoming query, and on high similarity to a cached query, return the cached answer without touching the model. The knob is the similarity threshold: too loose returns wrong answers, too tight never hits. Measure both rates on real traffic before trusting it.