Comparison
Distributed cachevsSemantic caching
Distributed cache
you move the cache into a shared service so every instance sees the same entries and one invalidation covers the whole fleet.
A cache held in a separate service such as Redis or Memcached, shared by every application instance. It gives you one copy to invalidate and a hit rate that does not degrade as you scale out. In exchange you add a network round trip to every lookup, a new dependency that can fail, and a serialisation format that is now part of your compatibility surface.
Full entry →Semantic caching
the second person asks roughly the same question in different words and you serve them the stored answer without calling the model.
Caching keyed on meaning: embed the incoming query, and on high similarity to a cached query, return the cached answer without touching the model. The knob is the similarity threshold: too loose returns wrong answers, too tight never hits. Measure both rates on real traffic before trusting it.
Full entry →