Paged Attention
Paged attention applies operating-system virtual memory paging to the KV Cache, and the whole idea is in that sentence. Instead of one contiguous VRAM block per request sized to the maximum output length, the cache is broken into small fixed-size pages (16 tokens each by default in vLLM). Pages live anywhere in GPU memory — non-contiguous, allocated on demand as the sequence actually grows — and a lightweight block table maps the logical page addresses the model sees to the physical page addresses where the data really sits. That indirection is the entire mechanism, and it dissolves all three modes of KV Cache Fragmentation at once: internal waste is bounded by one page, external gaps stop mattering because contiguity is no longer required, and duplication disappears because two requests' block tables can point at the same physical page.
That last property is the one that compounds. Because blocks are hashed by their token sequence, requests sharing a system prompt resolve to the same physical pages automatically — cross-request prefix sharing is not a separate feature bolted on, it is what content-addressed pages are. This is the mechanism underneath Prompt Caching, and it explains that concept's otherwise arbitrary rule that the stable content must come first: a cache hit is a chain of matching block hashes, and the first differing token ends the chain.
The transferable lesson is larger than LLM serving: a decades-old solution from an adjacent systems discipline often beats a novel domain-specific one. Nothing about paging is new; what was new was noticing that KV cache allocation had become a memory-management problem and that memory management was already solved.
Claims
- Non-contiguous, on-demand, fixed-size pages plus an indirection table eliminate fragmentation. principle — durable and not LLM-specific; it is virtual memory. That the technique transfers intact to GPU KV cache is the insight, not the technique.
- An indirection table makes physical sharing free: two logical pages can map to one physical page. principle — the property that turns fragmentation-avoidance into deduplication, and the reason prefix caching is a consequence of paging rather than an addition to it.
- Because blocks are hashed by token sequence, a cache hit requires an unbroken matching prefix. principle — the mechanism that grounds Prompt Caching's stable-prefix-first ordering rule.
- vLLM's default page size is 16 tokens. observation — engine default as reported by the source; version-dependent and check-worthy.
- A 13B model's weights (~26 GB) occupy ~65% of an A100 40 GB before any request arrives; the KV cache must fit in the remaining ~35%. observation — the source's arithmetic, framing why the KV budget is scarce. Check-worthy (the captions render the card as "A140 gigabyte").
- Tune
--gpu-memory-utilizationto 0.95 on stable workloads and 0.8 under bursty load. best practice — context: vLLM, whose default is 0.9; the flag sets what fraction of remaining VRAM goes to the KV cache. 0.95 is "best" only when the load shape is known and steady — it packs more concurrent requests but leaves no headroom; 0.8 is a concession to OOM under burst. The source's own guardrail is the durable part: benchmark your specific model before you commit (it names GuideLLM). A number copied without its load-shape context will OOM someone. - Enable prefix caching when many requests share a long prefix. best practice — context: RAG pipelines, multi-turn chat, and coding agents that re-send a fat system prompt; the source reports 75–95% hit rates for these. For one-shot heterogeneous prompts there is no shared prefix and the mechanism is moot.
Related
- KV Cache Fragmentation — the three-mode diagnosis that paging cures with one mechanism.
- KV Cache — what is being paged.
- Prefill and Decode Phases — paging makes decode's memory usable; prefix caching lets prefill be skipped entirely on a hit.
- Prompt Caching — the same cache reused across requests; paging is its mechanism, and block hashing is why its ordering rule exists.
- Speculative Decoding — the sibling lever, spending idle compute rather than reclaiming wasted memory.
- Distillate: KV Cache and Paged Attention: Why LLM Serving Is Memory-Bound