A pre-trained foundation model is optimized for text completion, not conversation, and its outputs can be factually or ethically problematic. Post-training fixes both in two steps. Supervised fine-tuning (SFT) optimizes the model for conversation using high-quality instruction/response data — teaching it what good responses look like. Preference fine-tuning then aligns the model with human values, classically via reinforcement learning from human feedback (RLHF): train a reward model that scores outputs by human preference, then optimize the model to maximize that score. Newer methods (direct preference optimization, DPO) are gaining traction, and some teams skip RL entirely — generating several outputs and keeping the highest-reward one, a strategy called best-of-N. This is also where the chat template is established: system/user prompts are combined by a model-specific template, and getting it even slightly wrong (an extra newline) causes silent quality failures.
Claims
- Pre-trained models are text completers; SFT converts them into instruction-followers. principle
- Preference tuning aligns outputs to human values by optimizing against a learned reward model (RLHF), with DPO and best-of-N as alternatives. observation
- SFT requires high-quality instruction data; preference tuning requires preference (winning/losing) data. observation
- Follow the model's chat template exactly — template mismatches (even an extra newline) cause silent failures. (best practice — context: input construction, especially when using third-party tooling to build prompts)
- Some teams skip the RL step and use best-of-N sampling against the reward model instead. observation
Related
- Decoding & Sampling Controls — best-of-N and sampling operate on the distribution post-training shapes.
- Parameter-Efficient Fine-Tuning (PEFT / LoRA) — the memory-efficient way to do the fine-tuning steps described here.
- Adaptation Strategy Ladder — post-training is the "fine-tune" rung applied by model providers before you ever adapt further.
- Data-Centric AI — the instruction/preference data quality that gates post-training.
- Distillate: AI Engineering in 76 Minutes — Chip Huyen's Book, Speedrun