Model distillation fine-tunes a small model to imitate a larger model's behavior, using data the large model generates on the target tasks. The payoff is that a small distilled model can outperform a larger general-purpose model on the specific tasks it was distilled for, at far lower inference cost. It underpins a practical adaptation recipe — the distillation path: start with a small dataset and the strongest model you can afford, train the best model you can with it, use that model to generate more training data, then use the expanded dataset to train a cheaper model. Distillation sits alongside quantization and pruning as a model-compression technique, but distinctively it changes which model you run rather than only how that model is represented.
Claims
- Distillation trains a small model to mimic a large one on target tasks using the large model's generated data. principle
- A small distilled model can outperform a larger general-purpose model on the distilled tasks. observation
- The distillation path — strong teacher on a small set → generate data → train a cheaper student — is a standard adaptation recipe. (best practice — context: producing a cheap task-specialized model when a strong but expensive model is available to teach it)
- Distillation is a model-compression technique alongside quantization and pruning. observation
Related
- Synthetic Data Generation — the teacher-generated data distillation depends on.
- Model Quantization — a compression sibling that shrinks representation rather than swapping the model.
- Model Merging — another route to one capable, cheaper model.
- Adaptation Strategy Ladder — distillation as a fine-tuning tactic on the ladder.
- Distillate: AI Engineering in 76 Minutes — Chip Huyen's Book, Speedrun