Batching combines multiple requests to process together, and it is one of the most powerful service-level inference optimizations because it raises hardware utilization and throughput. It comes in three forms of increasing sophistication. Static batching groups a fixed number of inputs but every request must wait until the batch is full — simple, but latency is inconsistent. Dynamic batching sets a maximum time window and processes the batch when it's either full or the window expires — giving more consistent latency guarantees. Continuous batching (a.k.a. in-flight batching) returns each response as soon as it completes and slots new requests in to keep the batch full — the best user experience, but the most complex to implement. Batching embodies the core serving trade-off: it improves throughput but can raise latency for individual requests, so the right form depends on the application's latency tolerance.
Claims
- Batching combines requests to raise utilization and throughput. principle
- Static / dynamic / continuous batching trade implementation complexity for latency consistency and responsiveness. observation
- Continuous batching gives the best user experience but is the hardest to implement. observation
- Batching improves throughput but can increase per-request latency — pick the form by the app's latency tolerance. (best practice — context: latency-sensitive vs throughput-sensitive serving)
Related
- Inference Latency Metrics — the latency/throughput axis batching trades along.
- Inference Bottlenecks — batching improves utilization against the binding bottleneck.
- Prefill and Decode Phases — decoupling prefill and decode is a complementary serving optimization.
- Prompt Caching — another serving-level optimization for repeated content.
- Distillate: AI Engineering in 76 Minutes — Chip Huyen's Book, Speedrun