KV Cache
In autoregressive generation the model emits one token at a time, but attention at every step must run over all previous tokens. Naively, generating a thousand-token response means the thousandth token recomputes the key and value matrices for all 999 tokens before it — work that is identical every time, because those tokens haven't changed. The KV cache stores the key and value matrices produced at each step so they are never recomputed: each new token computes only its own K, V, and Q, then attends over the cached history. This is the single technique that makes long-sequence generation practical.
It is not a free win, and the honest framing matters: the KV cache is a memory-for-compute trade. You stop paying quadratic recomputation and start paying VRAM that grows linearly with sequence length, per concurrent request. The trade is overwhelmingly worth it for long sequences — and it is exactly why serving is memory-bound rather than compute-bound at the decode step. Every downstream problem in LLM serving — fragmentation, OOM under burst, the concurrency ceiling — is the bill for this trade arriving. The cache is the solution that creates the next problem, which is what makes Paged Attention necessary and Prompt Caching valuable.
Claims
- Caching prior tokens' key/value matrices trades memory for compute. principle — durable: the recomputation it avoids is provably redundant (past tokens' K/V never change), and the cost is paid in VRAM proportional to sequence length × concurrent requests. Name it as a trade, not a free win — the memory side of the ledger is where serving actually fails.
- The trade is worth it for long sequences. best practice — context: autoregressive decode over sequences long enough that quadratic recomputation dominates. For a single very short generation the cache's allocation overhead earns little. Effectively always "best" in production serving, which is why it is a default rather than a knob.
- Optimizing serving means optimizing memory, not the model. principle — the diagnostic posture that follows: when latency climbs and throughput tanks under concurrency, the model is most likely not the culprit; how memory is used during inference is.
- Without a cache, the Nth generated token recomputes K and V for all N−1 tokens before it. observation — the concrete statement of the redundancy; the source's worked example uses N = 1000.
Related
- Prefill and Decode Phases — the cache is written during prefill and read every decode step; it is what makes decode memory-bound.
- Paged Attention — how the cache is allocated; without paging, the cache's memory cost is mostly waste.
- KV Cache Fragmentation — the specific ways naive KV cache allocation loses memory.
- Prompt Caching — reusing a KV cache prefix across requests rather than within one; the same cache, a wider reuse boundary.
- Context Compression — the complementary lever: shrink what must be cached, rather than cache it more efficiently.
- Distillate: KV Cache and Paged Attention: Why LLM Serving Is Memory-Bound