You can't optimize inference without measuring it, and for autoregressive models latency decomposes into two components: time to first token (TTFT) — how fast the first token appears after a query — and time per output token (TPOT) — how long each subsequent token takes. Total latency ≈ TTFT + TPOT × number of output tokens. A related metric, time to publish, accounts for the first token not always being shown to the user (e.g. when the model first generates a plan or chain-of-thought). Because latency varies request to request, percentiles are far more meaningful than averages. Beyond latency, throughput — output tokens per second across all requests — usually drives cost, so higher throughput means lower cost. The two are in tension: a fundamental latency-throughput trade-off means techniques like batching raise throughput but can raise individual latency, and your strategy must balance them for the application. Utilization metrics (model-FLOPs utilization, memory-bandwidth utilization) round out the picture.
Claims
- Latency = TTFT + TPOT × output tokens; TTFT and TPOT are separate levers. observation
- Report latency by percentiles, not averages, because it varies across requests. (best practice — context: measuring/optimizing user-facing inference)
- Throughput (tokens/sec across requests) drives cost; higher throughput lowers cost. observation
- There is a fundamental latency-vs-throughput trade-off; optimizing one can worsen the other. principle
- Time-to-publish differs from time-to-first-token when early output (a plan / reasoning) isn't shown to the user. observation
Related
- Inference Batching — the clearest instance of the latency/throughput trade.
- Inference Bottlenecks — what limits the metrics measured here.
- Total Cost of Inference — the cost side that throughput governs.
- Model Parallelism — a topology lever on latency and throughput.
- Distillate: AI Engineering in 76 Minutes — Chip Huyen's Book, Speedrun