Comparison
KV cachevsPrompt caching
KV cache
the long conversation got slower and then ran out of memory mid-generation, because the cache grows with every token in the window.
Memory holding the attention keys and values of all context tokens so each new token attends to history without recomputing it. It grows with context length and competes with weights for memory: the hidden reason long contexts get slow and tight setups crash mid-generation.
Full entry →Prompt caching
put static instructions and documents at the top, volatile user input at the bottom; caching works on prefixes.
Provider-side caching of a repeated prompt prefix (long system prompts, shared documents), cutting cost and TTFT substantially on cache hits. Anthropic exposes explicit cache control; OpenAI applies it automatically. Structure prompts with the stable part first to exploit it.
Full entry →