Research map / Large language models / Language, safety and interpretability

Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generation

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Statistical frameworks for LLM evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes top-H, an entropy-bounded truncated sampling method that greedily adds top tokens until the renormalized subset's entropy reaches alpha times the full distribution's entropy, improving creative-writing quality at high temperature.

Problem

Truncated sampling for LLMs trades off creativity (diversity) against coherence; existing methods like min-p use only the top-1 probability as a confidence proxy and ignore the overall distribution shape, causing over- or under-truncation, especially at high temperature.

Approach

Formulates an entropy-constrained minimum divergence (JSD) problem, shows it is equivalent to entropy-constrained mass maximization (ECMM), proves ECMM NP-hard, and solves it with a greedy algorithm that keeps the truncated distribution's entropy below alpha times the original entropy at each step.

Key results

• On GSM8K at T=2 with LLaMA3.1-8B-Instruct, top-H achieves up to 25.63% higher accuracy than min-p (39.35% vs 13.72%).
• On AlpacaEval, top-H improves win rate over min-p by up to 17.11% for LLaMA3.1-8B across temperatures.
• Top-H adds only about 0.8% per-token latency overhead relative to min-p and top-p in the reported timings.

Measured effects reported in the paper

Surprising findings

• Top-p collapses at high temperature (e.g., GSM8K accuracy 2.65% for LLaMA3.1-8B at T=2), while top-H degrades only modestly.
• The paper reports an empirical greedy-to-optimal mass ratio close to 1.0 on 20 Alpaca-Eval prompts, though no formal approximation guarantee is established for general distributions.

Limitations

• Only one hyperparameter alpha (set to 0.4 empirically); the greedy solution has no formal approximation guarantee beyond a Zipf-based heuristic analysis.
• Coherence is judged mainly by GPT-4o and log-likelihood proxies; no human evaluation in the main text.
• Evaluated on a small set of models (3B to 8B, plus a 70B in appendix) and a limited set of benchmarks.

Experimental setup

Datasets: AlpacaEval GSM8K GPQA MT-Bench

Benchmarks: AlpacaEval MT-Bench GSM8K GPQA

Metrics: length-controlled win rate judge score (1-10) exact match accuracy LLM-as-judge creativity and coherence scores per-token latency (ms/token)

Baselines: min-p sampling top-p sampling top-k sampling eta-sampling Mirostat

Builds on: min-p sampling top-p (nucleus) sampling eta-sampling Mirostat

Closest papers by meaning

• Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs (ICLR 2025 · similarity 0.92)
• Closing the Curious Case of Neural Text Degeneration (ICLR 2024 · similarity 0.89)
• Geometry-Aware Decoding with Wasserstein-Regularized Truncation and Mass Penalties for Large Language Models (ICML 2026 · similarity 0.89)
• Decoding Game: On Minimax Optimality of Heuristic Text Generation Strategies (ICLR 2025 · similarity 0.88)
• The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation (ICLR 2025 · similarity 0.88)
• XTC: Head-Aware Sampling by Excluding Top Choices (NeurIPS 2026 · similarity 0.88)