Research map / Learning theory and algorithms / Prediction and decision losses
On the Entropy Calibration of Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies whether LLM entropy calibration (entropy vs. log loss on human text) improves with scale, showing theoretically and empirically that miscalibration scales very slowly, and proposes a polynomial-time future-entropy adjustment that calibrates without raising log loss.
Problem
Autoregressive language models are entropy-miscalibrated: per-step entropy of generations grows with length while log loss on human text stays flat, causing error accumulation. Standard fixes (truncation, instruction tuning) reduce diversity by raising log loss; it is unclear whether miscalibration
Approach
Derives a scaling exponent for singleton (rare-token) mass under power-law data to predict that miscalibration scales slowly with model/data size, validates this empirically across model families, and proves a future-entropy adjustment (a first-order approximation of global temperature scaling) can calibrate entropy while not increasing log loss, given a black-box future-entropy regressor.
Key results
- Fitted scaling exponents are about 0 for text (WikiText 0.089 predicted; empirical ~0.0 to -0.13 for newer families) and about -0.33 to -0.35 for code, meaning miscalibration improves very slowly with scale for text.
- Instruction tuning and temperature reduction reduce entropy but increase log loss, so calibration comes at a diversity cost in the tested models.
- Theorem shows a future-entropy adjustment achieves |EntCE| <= 2T*delta + sum(1+alpha_t)*epsilon while not increasing log loss, assuming a black-box regressor with test error delta.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model scale | Base language models · WikiText-103 and WritingPrompts | calibration error: around 0.0; around -0.13 | No clear effect |
| Increase model scale | Base language models · CodeContests | calibration error: around -0.2; around -0.35 | Helped |
| Use a higher-quality, less diverse pretraining mixture with midtraining | Recent model families · WikiText-103 and WritingPrompts | calibration error: around 0.0 -> around -0.13 | Helped |
| Reduce sampling temperature below 1 | Base language models · WikiText-103, WritingPrompts, and CodeContests | entropy calibration error | Helped |
| Reduce sampling temperature below 1 | Base language models · WikiText-103, WritingPrompts, and CodeContests | log loss | Hurt |
| Apply instruction tuning | Base language models · WikiText-103, WritingPrompts, and CodeContests | entropy calibration error | Helped |
| Apply instruction tuning | Base language models · WikiText-103, WritingPrompts, and CodeContests | log loss | Hurt |
| Adjust token probabilities using predicted future entropy | Theoretical future entropy scaling procedure · Entropy calibration | log loss | Helped |
Surprising findings
- Larger instruction-tuned models can end up with entropy too low, showing post-training overshoots calibration.
- Miscalibration for text barely improves with model size: reducing calibration error 10x would require about 10^10 times more data at the fitted exponent.
Limitations
- The proposed calibration procedure is impractical (needs a separate future-entropy regressor per step and token, plus repeated sampling); no practical implementation is evaluated.
- Scaling exponents are fit on only three datasets and four model families, with training data mixtures not controlled, so the power-law link is not firmly established.
Experimental setup
Datasets: WikiText-103 WritingPrompts CodeContests
Benchmarks: WikiText-103 WritingPrompts CodeContests
Metrics: entropy calibration error log loss entropy per step scaling exponent
Baselines: temperature scaling top-k sampling top-p sampling min-p sampling instruction-tuned models
Builds on: entropy calibration (Cao et al.) temperature scaling MaxEnt RL Pythia top-p sampling
Closest papers by meaning
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.88)
- Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training (NeurIPS 2025 · similarity 0.88)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.88)
- Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (NeurIPS 2025 · similarity 0.88)
- Calibrated Language Models and How to Find Them with Label Smoothing (ICML 2025 · similarity 0.87)
- Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.