Research map / Large language models / Reasoning and chain of thought

Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes token entropy in CoT and RLVR, then restricts DAPO policy-gradient updates to the top 20% highest-entropy ("forking") tokens, matching or beating full-token training on Qwen3 models.

Problem

RLVR for LLM reasoning trains on all tokens, with limited understanding of which tokens actually drive reasoning gains and how RLVR changes the model's entropy patterns.

Approach

Token entropy reveals a minority of high-entropy "forking" tokens that steer reasoning; RLVR largely preserves base-model entropy patterns and mainly changes high-entropy tokens, so restricting policy gradients to the top 20% highest-entropy tokens per batch improves RLVR, with gains growing with model size.

Key results

• Using only the top 20% highest-entropy tokens improves Qwen3-32B DAPO by +7.71 on AIME'24 (63.54 vs 55.83) and +11.04 on AIME'25 (56.67 vs 45.63).
• On Qwen3-14B, the top-20% variant gains +5.21 on AIME'24 and +4.79 on AIME'25; on Qwen3-8B performance is roughly unchanged (+1.25 and +0.83).
• Training only on the bottom 80% lowest-entropy tokens leads to a marked performance decline; extending max response length to 29,696 raises the 32B AIME'24 score to 68.12.

Measured effects reported in the paper

Surprising findings

• Training on only 20% of tokens matches or exceeds full-token DAPO, going beyond the 80/20 rule.
• Base model's top-20% entropy positions retain about 86.67% overlap with the final RLVR model, indicating RLVR largely preserves base entropy patterns.

Limitations

• Experiments mainly on Qwen family; Llama-3.1-8B results weak on AIME, so less convincing.
• Evaluated only on math benchmarks plus one out-of-distribution code benchmark (LiveCodeBench).
• Optimal 20% fraction may need tuning in different RLVR settings.

Experimental setup

Datasets: DAPO-Math-17K AIME'24 AIME'25 AMC'23 MATH500 Minerva OlympiadBench LiveCodeBench

Benchmarks: AIME'24 AIME'25 AMC'23 MATH500 Minerva OlympiadBench LiveCodeBench

Metrics: average accuracy over 16 samples (Acc@16) average response length (Len@16) overall entropy

Baselines: DAPO with all tokens

Builds on: DAPO GRPO PPO verl

Closest papers by meaning

• Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.92)
• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.92)
• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.92)
• Well Begun, Half Done: Reinforcement Learning with Prefix Optimization for LLM Reasoning (AAAI 2026 · similarity 0.91)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.91)
• The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning (NeurIPS 2025 · similarity 0.91)