Applied AI·Local and self-hosted inference
the long conversation got slower and then ran out of memory mid-generation, because the cache grows with every token in the window.
KV cache
Draft summary, pending review
Memory holding the attention keys and values of all context tokens so each new token attends to history without recomputing it. It grows with context length and competes with weights for memory: the hidden reason long contexts get slow and tight setups crash mid-generation.