Research map / Large language models / Language, safety and interpretability
Improving Open-Ended Text Generation via Adaptive Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes adaptive decoding, which dynamically sizes the candidate token set per step by thresholding an entropy-based "confidence" increment, and evaluates it on open-ended generation and math reasoning with GPT2-XL and Llama2 models.
Problem
Open-ended text generation with language models needs a sensible next-token candidate set: fixed top-k or top-p truncation either induces repetition (too small) or incoherence and hallucination (too large).
Approach
Treats choosing the candidate set as a confidence-increasing process: tokens are moved in descending probability order from an unknown pool to the candidate set, and a token is kept while its normalized entropy-based confidence increment (Conf_k, min-max scaled entropy with a pivot k) stays above a threshold epsilon.
Key results
- On GPT2-XL, adaptive decoding achieves the best diversity (89.30 on WritingPrompts vs 80.50 for top-k) and best MAUVE among stochastic methods on WikiText-103 (79.57 vs 78.22 for top-p), while coherence is lower than top-k on WikiText-103 (55.98 vs 61.19).
- On MT-Bench with Llama2-7B-chat judged by GPT-4, adaptive decoding obtains a mean score of 6.4375 versus 6.2656 for top-p.
- On GSM8K zero-shot with Llama2-70B, adaptive decoding reaches 58.67 accuracy versus 56.00 for top-p and 56.33 for greedy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adaptive decoding instead of other decoding methods | GPT2-XL · WritingPrompts and WikiText-103 | Diversity and MAUVE: WritingPrompts diversity 89.30* and MAUVE 25.18*; WikiText-103 diversity 93.23* and MAUVE 79.57 | Helped |
| use adaptive decoding instead of other decoding methods | Llama2-7B · WritingPrompts | MAUVE: 35.26* | Helped |
| use adaptive decoding instead of other decoding methods | Llama2-7B · WritingPrompts | Diversity: adaptive 89.98; η-sampling 90.99 | Mixed |
| use adaptive decoding instead of other decoding methods | Llama2-7B · WikiText-103 | MAUVE: adaptive 76.74; η-sampling 78.26 | Mixed |
| use adaptive decoding instead of Top-p | Llama2-7B-chat · WritingPrompts | Diversity: 87.41 -> 93.99 | Helped |
| use adaptive decoding instead of Top-p | Llama2-7B-chat · WritingPrompts | Coherence: 49.85 -> 54.39 | Helped |
| use adaptive decoding instead of Top-p | Llama2-7B-chat · MT-Bench | GPT-4 score: 6.2656 -> 6.4375 | Helped |
| use adaptive decoding instead of Top-p | Llama2 chat models · GSM8K | accuracy: 29.00 -> 30.00; 37.00 -> 40.33; 56.00 -> 58.67 | Helped |
| use adaptive decoding instead of Top-p | Llama2 chat models · MultiArith | accuracy: 68.33 -> 69.33; 73.33 -> 71.33; 85.33 -> 87.00 | Mixed |
| use adaptive decoding rather than human-written continuations | Llama2-7B · WikiText-103 | human preference: adaptive preferred 4.000%; human preferred 51.000*% | Hurt |
| use adaptive decoding instead of Top-k | Llama2-7B · WikiText-103 | human preference: 64.500*% preferred adaptive; 18.500% preferred Top-k | Helped |
| increase threshold ε from 0.0005 to 0.001 (ablation) | GPT2-XL adaptive decoding · WritingPrompts | MAUVE: 47.19 -> 55.19 | Helped |
Surprising findings
- Expanding the candidate set improved reasoning accuracy for smaller Llama2 chat models but hurt larger ones, while adaptive decoding maintained performance.
- Mirostat achieves high coherence largely through repetition, so its coherence score is misleading without diversity and MAUVE.
Limitations
- Adaptive decoding adds computational overhead, with decoding latency roughly 1.4-1.8x that of top-p in the reported measurements.
- Reasoning evaluation is limited to small zero-shot sets (300 samples per dataset) and the authors state that more extensive reasoning experiments are needed.
- Coherence is lower than several baselines on WikiText-103 and the approach is only verified on language models, not multimodal settings.
Experimental setup
Datasets: WikiText-103 WritingPrompts GSM8K MultiArith MT-Bench
Benchmarks: WikiText-103 WritingPrompts MT-Bench GSM8K MultiArith
Metrics: rep-2 rep-3 rep-4 diversity MAUVE coherence (SimCSE) GPT-4 score accuracy human preference win rate
Baselines: greedy decoding top-k top-p mirostat typical decoding eta-sampling
Builds on: top-p sampling top-k sampling eta-sampling typical decoding mirostat
Closest papers by meaning
- Reflection-Window Decoding: Text Generation with Selective Refinement (ICML 2025 · similarity 0.90)
- Entropy-informed Decoding: Adaptive Information-Driven Branching (ICML 2026 · similarity 0.89)
- Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Models (AAAI 2024 · similarity 0.88)
- Forking Paths in Neural Text Generation (ICLR 2025 · similarity 0.88)
- Decoding Game: On Minimax Optimality of Heuristic Text Generation Strategies (ICLR 2025 · similarity 0.88)
- Reranking Laws for Language Generation: A Communication-Theoretic Perspective (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.