When compute is the limiting factor in training a foundation model, the Chinchilla scaling law gives the compute-optimal split between model size and dataset size for a fixed compute budget. Its headline heuristic: the number of training tokens should be roughly 20× the number of parameters — so a 3B-parameter model wants on the order of 60B training tokens. It reframes "bigger model" as "the wrong question": for a given budget, an over-large model starved of data is a worse use of compute than a smaller model trained on proportionally more tokens. The law also frames the field's ceiling — parameter count estimates the compute needed for training and inference, but the cost of marginal quality keeps rising (going from a 3% to a 2% error rate can take an order of magnitude more data/compute/energy), and scaling runs into two hard limits: running out of high-quality internet text, and data-center electricity.
Claims
- Compute-optimal training puts training tokens at roughly 20× parameter count. (observation — the source's rendering of the Chinchilla result; a groundable number for a later check)
- For a fixed compute budget, compute — not data or parameters alone — is the binding constraint the law optimizes against. principle
- The cost of equal quality improvements stays high even as the cost of a fixed quality level falls over time. observation
- Parameter count can mislead for sparse models (many zeros): a large sparse model may need less compute than a smaller dense one. observation
- Continued scaling faces two bottlenecks: exhausting high-quality training text, and data-center electricity (already ~1–2% of global supply, per the source). (observation — source's claim)
Related
- Decoding & Sampling Controls — the other half of "how models behave": how outputs are sampled once the model is trained.
- Perplexity & Cross-Entropy — the intrinsic metric that tracks how well a model has learned its training distribution.
- Model Quantization — a post-hoc lever on the memory/compute cost that scaling drives up.
- Data-Centric AI — the response to the data-quality ceiling this law runs into.
- Distillate: AI Engineering in 76 Minutes — Chip Huyen's Book, Speedrun