A foundation model does not emit a single definitive answer; it produces a probability distribution over next tokens, and how you sample from that distribution materially changes the output. Greedy sampling always takes the highest-probability token — fast and deterministic, but repetitive and flat. To introduce variation you reach for sampling controls: temperature scales the model's confidence (higher ≈ 0.7–1 is more creative and less accurate; near-zero is deterministic and focused); top-k restricts the choice to the k most likely tokens (typically 50–500); top-p / nucleus keeps the smallest set of tokens whose cumulative probability clears a threshold p (0.9 ≈ the tokens making up 90% of the mass). The load-bearing point is not the knob values but the framing: because generation is probabilistic, behaviors like inconsistency under minor input changes and confident hallucination are inherent, not bugs to be fully eliminated.
Claims
- Model output is sampled from a next-token probability distribution, so the sampling method changes the response. principle
- The probabilistic nature of sampling is the direct cause of output inconsistency and hallucination. principle
- Greedy decoding maximizes likelihood but yields repetitive, predictable text. observation
- Higher temperature trades accuracy for creativity; near-zero temperature does the reverse. (best practice — context: tune toward determinism for factual/structured tasks, toward higher temperature for open-ended generation)
- top-k (fixed count) and top-p (cumulative-probability) both cap the sampling pool; typical ranges are k≈50–500 and p≈0.9. observation
Related
- Perplexity & Cross-Entropy — the training-time view of the same distribution the sampler draws from.
- LLM-as-Judge — judges are themselves probabilistic, so the same input can score differently on re-run.
- Chinchilla Scaling Law — governs how the underlying distribution is learned.
- Distillate: AI Engineering in 76 Minutes — Chip Huyen's Book, Speedrun