Model merging combines the weights of several models into one — as opposed to ensembling, which combines their outputs. Because the result is a single model, it captures the benefit of multiple specialists without the higher inference cost of running several models at once. If two models excel at different aspects of the same task, merging can produce one model that beats both. Approaches range in sophistication: summing (adding the constituent models' weight values — the most common), layer stacking (taking different layers from different models, also called franken-merging), and concatenation (combining parameters — less recommended, since it doesn't reduce memory versus keeping the models separate). Merging can often be done without GPUs, can improve performance while shrinking memory footprint, suits on-device deployment, and can support federated learning where devices train on separate data.
Claims
- Merging combines model weights; ensembling combines model outputs. principle
- Merging avoids the higher inference cost of ensembling by producing a single model. observation
- Merging two models strong at different aspects of a task can outperform both. observation
- Summing weights is the most common merge; layer-stacking (franken-merging) and concatenation are alternatives, with concatenation least recommended as it doesn't cut memory. observation
- Merging can run without GPUs, reduce memory footprint, and enable on-device and federated setups. (best practice — context: multi-task or on-device deployment where running multiple models is too costly)
Related
- Parameter-Efficient Fine-Tuning (PEFT / LoRA) — the per-task fine-tuning whose outputs merging combines.
- Model Distillation — an alternative way to get one capable small model.
- Model Quantization — a complementary compression lever for on-device targets.
- Distillate: AI Engineering in 76 Minutes — Chip Huyen's Book, Speedrun