To optimize inference you first have to know what's slowing it down, and AI workloads face two distinct bottlenecks. Compute-bound workloads are limited by available computational power — intensive-calculation tasks like image generation are typically compute-bound. Memory-bandwidth-bound workloads are limited by how fast data moves between memory and processors — and autoregressive language-model inference is typically memory-bandwidth-bound, because generating each token re-reads large weight and KV tensors. The distinction is load-bearing because different optimizations fix different bottlenecks: a compute-bound workload benefits from more powerful chips or distributing work across chips, while a memory-bandwidth-bound workload benefits from chips with higher memory bandwidth (and from techniques like quantization that move less data). Profiling tools (e.g. NVIDIA Nsight) identify which bottleneck applies via a roofline chart.
Claims
- AI workloads are limited by either compute or memory bandwidth; the two need different fixes. principle
- Autoregressive LLM inference is typically memory-bandwidth-bound. principle
- Compute-bound work wants more/faster chips; memory-bandwidth-bound work wants higher memory bandwidth. (best practice — context: hardware selection and optimization once the bottleneck is known)
- Profile before optimizing — a roofline chart identifies which bottleneck you have. (best practice — context: don't guess the bottleneck; measure it)
- For inference hardware, focus on FLOPs, memory size, and memory bandwidth, weighted by your bottleneck. observation
Related
- Model Quantization — reduces data moved, attacking the memory-bandwidth bound directly.
- Inference Batching — a service-level lever on utilization and throughput.
- Model Parallelism — how to distribute a compute- or memory-bound model across machines.
- Inference Latency Metrics — the metrics that reveal a bottleneck's user-facing cost.
- KV Cache — the tensor whose repeated reads drive the memory-bandwidth bound.
- Distillate: AI Engineering in 76 Minutes — Chip Huyen's Book, Speedrun