Research map / Large language models / Language, safety and interpretability
Closing the Curious Case of Neural Text Degeneration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a theoretical account of why threshold truncation sampling works (log-probability underestimation bounds) and proposes basis-aware threshold (BAT) sampling, which uses the softmax bottleneck via linear programs to certify tokens in the true support, evaluated with GPT-2 on OpenWebText.
Problem
Why truncation sampling heuristics such as nucleus, eta and epsilon sampling work so well in language generation is not theoretically understood, and threshold truncation cannot separate low-quality high-probability tokens from high-quality low-probability tokens.
Approach
Shows that log-probability underestimation caused by the low-rank softmax bottleneck yields an additive bound on probability overestimation, justifying thresholds; then uses the softmax matrix's linear constraints (W^T p = W^T p_hat) in a linear program to prove tokens have nonzero true probability without a fixed threshold.
Key results
- BA-eta outperforms eta sampling in MAUVE for all four GPT-2 sizes at low-entropy settings (e.g., GPT-2 Small 87.8 vs 85.0, XL 89.6 vs 87.1).
- In pairwise human evaluation, BA-eta narrowly beats eta at one setting (0.43 vs 0.38 wins) but is roughly tied at another (0.48 vs 0.47).
- Estimated rank of truncated log-probability matrices keeps growing past GPT-2 XL's hidden size of 1600, while the untruncated matrix saturates at 1600.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BA-η instead of η for lower-entropy generation | GPT-2 · Open Web Text test set, GPT-2 Small | MAUVE: 85.0_{1.4} -> 87.8_{1.0} | Helped |
| Use BA-η instead of η for lower-entropy generation | GPT-2 · Open Web Text test set, GPT-2 Medium | MAUVE: 90.4_{0.1} -> 92.2_{0.6} | Helped |
| Use BA-η instead of η for lower-entropy generation | GPT-2 · Open Web Text test set, GPT-2 Large | MAUVE: 86.0_{0.5} -> 88.4_{0.5} | Helped |
| Use BA-η instead of η for lower-entropy generation | GPT-2 · Open Web Text test set, GPT-2 XL | MAUVE: 87.1_{1.2} -> 89.6_{0.4} | Helped |
| Use BA-η instead of η | GPT-2 · Human pairwise evaluation, BA-η matched to η=0.002 | win rate: 0.38 -> 0.43 | Helped |
| Use BA-η instead of η | GPT-2 · Human pairwise evaluation, BA-η matched to η=0.024 | win rate: 0.47 -> 0.48 | No clear effect |
| Increase the number of basis-aware constraints (ablation) | BA-η sampling · Open Web Text test set, GPT-2 XL | MAUVE | Helped |
| Use BA-nucleus instead of nucleus sampling (ablation) | GPT-2 · Open-ended generation | MAUVE | Hurt |
| Use BAT methods rather than their vanilla threshold-based counterparts | GPT-2 · Open Web Text test set across model sizes | MAUVE | Mixed |
| Use BA-η rather than η at more restrictive, lower-entropy settings | GPT-2 across model sizes · Open Web Text test set, low-entropy decoding | MAUVE | Helped |
Surprising findings
- BA-nucleus performed significantly worse than BA-eta and BA-epsilon in preliminary experiments.
- BA-epsilon accepts no tokens for certain prefixes at sufficiently low-entropy parameters, and the method has no clear fallback to greedy decoding.
Limitations
- Linear program solving remains computationally expensive, making exact candidate-set computation over the full vocabulary infeasible.
- BAT pairs poorly with nucleus sampling and can accept no tokens for some prefixes at low-entropy settings, with no principled fallback to greedy decoding.
- Pilot study only uses GPT-2 models and mostly MAUVE; human gains are narrow.
Experimental setup
Datasets: OpenWebText
Benchmarks: OpenWebText
Metrics: MAUVE pairwise human preference win rate
Baselines: eta sampling epsilon sampling nucleus sampling
Builds on: nucleus sampling epsilon sampling eta sampling softmax bottleneck
Closest papers by meaning
- Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs (ICLR 2025 · similarity 0.90)
- Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generation (NeurIPS 2025 · similarity 0.89)
- Fewer Truncations Improve Language Modeling (ICML 2024 · similarity 0.88)
- The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation (ICLR 2025 · similarity 0.88)
- Error Norm Truncation: Robust Training in the Presence of Data Noise for Text Generation Models (ICLR 2024 · similarity 0.88)
- Decoding Game: On Minimax Optimality of Heuristic Text Generation Strategies (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.