Fine-tuning's dominant cost is memory, and memory scales with the number of trainable parameters — so the central move of parameter-efficient fine-tuning (PEFT) is to update only a tiny inserted set of parameters while freezing the rest. PEFT methods split into adapter-based (additive) methods that add new weights and soft-prompt methods that introduce trainable tokens. The most popular is LoRA (low-rank adaptation): for a weight matrix of size n×m, LoRA picks a small rank r and learns two matrices (n×r and r×m); only those are updated, the original weights stay frozen, and at inference the two can be multiplied and merged back into the original weights — so, unlike classic adapters, LoRA adds no inference latency. PEFT is not only parameter-efficient but sample-efficient (hundreds to a few thousand examples versus the millions full fine-tuning can need), and adapter methods let you serve many task variants that share one base model.
Claims
- Memory footprint scales with trainable parameter count; freezing most parameters is the core PEFT lever. principle
- LoRA decomposes weight updates into two low-rank matrices, trains only those, and merges them back at inference — adding no inference latency. observation
- PEFT is both parameter- and sample-efficient: it can work with hundreds to a few thousand examples where full fine-tuning may need millions. observation
- Start with adapter techniques like LoRA before attempting full fine-tuning. (best practice — context: beginners and memory-constrained setups)
- Adapter methods let multiple fine-tuned variants share a single base model. observation
- Key hyperparameters — learning rate, batch size, epochs, prompt-loss-weight — significantly affect results; smaller datasets need more epochs. (best practice — context: tuning a fine-tuning run; e.g. 4–10 epochs for thousands of examples, 1–2 for millions)
Related
- Model Quantization — the other main memory-reduction lever; often combined with PEFT.
- Model Merging — combining separately-adapted models, complementary to adapters.
- Post-Training Alignment — the fine-tuning stages PEFT makes affordable.
- Adaptation Strategy Ladder — PEFT is how the fine-tune rung is taken in practice.
- Distillate: AI Engineering in 76 Minutes — Chip Huyen's Book, Speedrun