Quantization converts a model from a higher-bit numeric format to a lower-bit one to cut memory and speed up inference. The arithmetic is direct: a 13B-parameter model in fp32 needs ~4 bytes per parameter ≈ 52 GB; drop to 16-bit and it's ~26 GB. Inference is typically run at as few bits as tolerable (16, 8, even 4); training is more sensitive to precision and is usually done in mixed precision — some operations in higher precision (32-bit), others in lower (16/8-bit). Numeric formats trade range (the span of representable values) against precision (how finely a value can be represented), and reducing precision can shift values or introduce errors — which is why a model must be loaded in its intended format. Weight-only quantization is by far the most popular compression technique: easy to implement, works out of the box for many models, and delivers large savings for little effort.
Claims
- Memory ≈ parameters × bytes-per-parameter, so halving bit-width roughly halves memory (e.g. 13B: 52 GB fp32 → 26 GB fp16). (observation — the source's arithmetic; groundable)
- Inference tolerates low precision (down to 4-bit); training needs mixed precision because it is precision-sensitive. principle
- Numeric formats balance range against precision; these are distinct axes. principle
- Load a model in its intended format — e.g. LLaMA 2 weights were tuned for bf16 and gave significantly worse quality loaded as fp16. (best practice — context: deploying/serving a released model; format mismatch silently degrades quality)
- Weight-only quantization is the most popular compression method for its ease and payoff. observation
Related
- Parameter-Efficient Fine-Tuning (PEFT / LoRA) — the complementary memory lever (fewer trainable params vs fewer bits).
- Inference Bottlenecks — quantization mainly attacks the memory-bandwidth bound.
- Model Distillation — another model-compression route, listed alongside quantization and pruning.
- Total Cost of Inference — the cost surface quantization is trying to move.
- Distillate: AI Engineering in 76 Minutes — Chip Huyen's Book, Speedrun