Prefill and Decode Phases
LLM inference is not one workload but two, with opposite bottlenecks. Prefill ingests the whole prompt at once — every token runs through every transformer layer to build the representation the model needs before it can emit a single output token. Because the entire prompt is available simultaneously, prefill is a big dense matrix multiply: it is compute-bound, and it is the lag you feel before the first token appears (time-to-first-token). Decode then emits one token at a time; each step is a small matrix multiply against a large cached context that must be read out of GPU memory. Arithmetic intensity collapses, and decode becomes memory-bound — the GPU spends most of each step waiting on VRAM reads rather than computing. This split is not an implementation detail of any engine; it falls out of the shape of autoregressive attention. It is also the reason every other serving optimization exists: KV Cache makes decode tractable, Paged Attention makes decode's memory usable, and Speculative Decoding spends the compute that decode leaves idle.
The practical consequence is that a serving stack running one phase at a time under-uses the hardware in both. A prefill running to completion starves active decode streams (visible as stuttering token output when a long prompt arrives); a batch of pure decode leaves compute units idle between memory reads. Chunked prefill is the direct answer: prioritize decode requests in the batch, then fill the remaining compute budget with chunks of pending prefill.
Claims
- Inference has two phases with opposite bottlenecks: prefill is compute-bound, decode is memory-bound. principle — durable and architectural; it follows from autoregressive attention (all prompt tokens at once, then one token at a time), not from any engine's design.
- The lag before the first token is prefill; the fast stream afterwards is decode. observation — the user-visible signature of the split, and the reason TTFT and inter-token latency are separate metrics that respond to separate levers.
- Because decode re-reads the full context from VRAM at every step, memory — not the model — is the bottleneck in LLM serving. principle — the load-bearing consequence: if a deployment degrades under concurrency, look at memory before reaching for a smaller model.
- Enable chunked prefill for throughput-heavy workloads. best practice — context: serving where long prompts arrive alongside active decode streams and cause visible stutter, because the engine (vLLM by default) runs prefill to completion before resuming decode. It is "best" because it interleaves a compute-bound phase with a memory-bound one so neither unit idles. Not indicated when a single stream's latency is all that matters and there is no batch to interleave with.
- Production deployments have seen ~50% throughput improvement
from chunked prefill, paired with
max_num_batched_tokens > 2048. observation — attributed to the source (IBM Technology, on vLLM); a check-worthy figure and a version-dependent default.
Related
- KV Cache — what decode reads on every step; the cache exists to make decode cheap, and in doing so is what makes decode memory-bound.
- Paged Attention — how the memory that decode reads is allocated; the fix for the waste that the memory-bound phase exposes.
- Speculative Decoding — exploits the compute that the memory-bound decode phase leaves idle.
- KV Cache Fragmentation — the failure mode of decode's memory when it is allocated naively.
- Prompt Caching — skipping prefill entirely when its result is already cached; the API-surface view of the same phase boundary.
- Distillate: KV Cache and Paged Attention: Why LLM Serving Is Memory-Bound