As models grow, a single machine may not suffice, so work is distributed across machines. The simplest form is replica parallelism: make multiple copies of the model, each handling different requests — ideal for high-throughput scenarios. When a single model won't fit on one machine, model parallelism splits the model itself: tensor parallelism breaks individual operations into smaller pieces; pipeline parallelism divides the model into sequential stages; context parallelism splits input sequences across devices; and sequence parallelism splits different operations across machines. Which combination to use depends on workload and performance requirements — for low-latency-priority applications, replica parallelism may be best despite higher cost. Across most use cases the highest-impact techniques are quantization, tensor parallelism, replica parallelism, and attention-mechanism optimization.
Claims
- Replica parallelism (multiple full copies) is the simplest approach and suits high throughput. observation
- Model parallelism splits a single model via tensor, pipeline, context, or sequence parallelism when it won't fit on one machine. observation
- The optimal parallelism combination depends on workload and performance requirements; replica parallelism can win for low-latency priorities despite cost. (best practice — context: choosing a serving topology under a latency vs cost priority)
- The most impactful serving techniques overall are quantization, tensor parallelism, replica parallelism, and attention-mechanism optimization. observation
Related
- Inference Bottlenecks — parallelism is chosen against the binding bottleneck.
- Inference Batching — the intra-machine throughput lever, complementary to cross-machine parallelism.
- Model Quantization — co-listed as a top-impact serving technique.
- Inference Latency Metrics — the latency/throughput targets parallelism serves.
- Distillate: AI Engineering in 76 Minutes — Chip Huyen's Book, Speedrun