Speculative Decoding
Decode is memory-bound: at each step the GPU reads a large cached context out of VRAM and does comparatively little arithmetic with it, so the compute units sit idle waiting on memory. Speculative decoding spends that idle compute. A small draft model proposes a run of several output tokens; the large model then verifies all of them in a single forward pass, accepting the prefix of proposals it agrees with and correcting the first one it doesn't. Because verification is one batched forward pass over several positions — precisely the compute-dense shape the memory-bound step was wasting — several tokens can emerge for roughly the cost of one.
The property that makes this safe rather than a quality/speed trade is that the accept/reject rule is constructed so the output distribution is identical to the large model's; you are not approximating the big model, you are pipelining it. The property that makes it contextual is that it consumes spare compute — and at high concurrency there is none spare, because a full batch already saturates the GPU. Speculative decoding is therefore a latency technique, not a throughput technique, and its gains shrink exactly as your utilization improves. Reach for it when interactive latency matters more than raw throughput; a variant with no draft model at all (an n-gram speculator, which proposes continuations by matching against text already in context) costs nothing and helps on structured or repetitive output.
Claims
- Memory-bound decode leaves compute idle; speculation converts idle compute into tokens. principle — durable and mechanical: it follows from the arithmetic intensity of single-token decode, and it is why the technique exists at all rather than being a heuristic.
- A draft model proposes k tokens; the large model verifies them in one forward pass. observation — the mechanism as the source describes it: accepted tokens ship, the first wrong one is corrected.
- Output quality is mathematically identical to running the large model alone. observation — the source asserts this plainly. It is the advertised property of rejection-sampling verification; attributed here as the source's claim and left open for a targeted check, not adjudicated.
- Reach for speculative decoding when interactive latency matters more than raw throughput. best practice — context: low-to-moderate concurrency, where the GPU has spare compute between memory reads. The source is unusually honest that gains shrink at very high concurrency because the batch already keeps the GPU busy. Invert the context and the technique becomes pure overhead — a draft model burning the compute a full batch needed.
- vLLM ships a zero-cost n-gram speculator
(
--speculative-model ngram) for structured or repetitive outputs. observation — attributed to the source; a version-dependent flag and check-worthy.
Related
- Prefill and Decode Phases — the memory-bound decode step whose idle compute this technique spends; chunked prefill is the other way to fill it.
- KV Cache — the cached context whose VRAM reads make decode memory-bound in the first place.
- Paged Attention — the sibling lever: reclaim wasted memory rather than wasted compute.
- Model-Tier Routing — a related use of a small model alongside a large one, but with the opposite guarantee: routing accepts a quality change to save cost, speculation refuses one.
- Distillate: KV Cache and Paged Attention: Why LLM Serving Is Memory-Bound