Perplexity and cross-entropy are the intrinsic metrics that measure how well an autoregressive language model predicts the next token. Entropy is how much information, on average, a token carries — higher entropy means each token is more surprising. Cross-entropy measures how far the model's predicted distribution is from the data's true distribution; a perfectly trained model reaches cross-entropy equal to the data's entropy (KL divergence zero). Perplexity is simply the exponential of cross-entropy — the effective number of options the model is choosing among when it predicts. What counts as "good" perplexity depends entirely on the data (more structured data is more predictable → lower perplexity; larger vocabulary → higher; longer context → lower). Its value is as a proxy for general capability and as a diagnostic — but it degrades as a quality signal after heavy post-training, since a model that got better at tasks can get statistically worse at raw next-token prediction.
Claims
- Perplexity is the exponential of cross-entropy; both measure next-token predictive quality. observation
- A perfectly trained model's cross-entropy equals the entropy of its training data. principle
- "Good" perplexity is data-dependent — set by structure, vocabulary size, and context length, not an absolute bar. principle
- Perplexity becomes a less reliable quality signal after SFT/RLHF post-training. observation
- Unusually low perplexity on eval data can flag training-data contamination; unusually high perplexity can flag nonsensical text. (best practice — context: contamination and gibberish detection, e.g. checking a benchmark wasn't in the training set)
Related
- Levels of Evaluation — perplexity is the intrinsic layer beneath task- and business-level evaluation.
- Decoding & Sampling Controls — the inference-time counterpart to this training-time metric.
- LLM-as-Judge — the subjective-evaluation counterpart when exact metrics don't apply.
- Distillate: AI Engineering in 76 Minutes — Chip Huyen's Book, Speedrun