Research map / Large language models / Reasoning and chain of thought
The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that pure entropy minimization, without labels, via unsupervised finetuning (EM-FT), entropy-only RL (EM-RL), or inference-time logit optimization (EM-INF) improves LLM reasoning on math, coding, and science tasks.
Problem
Whether LLMs' reasoning can be improved using entropy minimization alone without labeled data, during post-pretraining adaptation or at inference time.
Approach
Three label-free entropy-minimization methods mirror post-pretraining stages: EM-FT directly minimizes token-level entropy on model samples like SFT, EM-RL uses negative entropy as the only RL reward, and EM-INF optimizes output logits at each decoding step to reduce entropy without any parameter updates.
Key results
- EM-RL-token on Qwen2.5-Math-7B without labels averages 42.9 on math vs 42.1 for labeled GRPO (N=4), and EM-RL-sequence reaches 27.0/26.4 on coding vs 25.4 for GRPO.
- EM-INF with Qwen2.5-32B-Instruct matches or exceeds GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro on SciCode main problems, improving over its base model by about 6%.
- EM-INF averages 46.4 on math vs 45.4 for self-consistency (N=4) on Qwen2.5-Math-7B-Instruct, with AIME rising to 14.4 from 11.1, at roughly one-third the FLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use negative token-level entropy as the RL reward | Qwen2.5-7B EM-RL · Coding average | score: 22.3 -> 27.0 | Helped |
| Use negative sequence-level entropy as the RL reward | Qwen2.5-7B EM-RL · LeetCode | score: 26.1 -> 31.1 | Helped |
| Minimize token-level entropy on unlabeled model outputs | Qwen2.5-7B EM-FT · LiveCodeBench-v2 | score: 18.4 -> 17.2 | No clear effect |
| Optimize logits to minimize entropy at inference | Qwen2.5-Math-7B-Instruct · AIME | score: 11.1 -> 14.4 | Helped |
| Optimize logits to minimize entropy at inference | Qwen2.5-7B-Instruct · UGPhysics | score: 23.4 -> 26.9 | Helped |
| Optimize logits to minimize entropy at inference | Llama-3.1-8B-Instruct · Math average | score: 19.6 -> 21.7 | Hurt |
| Apply entropy minimization to individualistic value reasoning | Qwen2.5-7B-Instruct · IndVal | accuracy: 65.9 -> 66.0 | No clear effect |
| Add KL regularization to EM-FT with N=1 (ablation) | Qwen2.5-7B EM-FT · Math average | score: 40.2 -> 39.6 | Hurt |
| Add KL regularization and use N=4 trajectories (ablation) | Qwen2.5-7B EM-FT · Math average | score: 40.2 -> 41.7 | Helped |
| Increase EM-FT sampling from N=1 to N=4 while adding KL regularization (ablation) | Qwen2.5-7B EM-FT · Coding average | score: 22.8 -> 21.8 | Hurt |
Surprising findings
- Entropy minimization with only negative entropy reward (no labels) matches or beats labeled GRPO and RLOO on several math and coding benchmarks for Qwen2.5-based models.
- EM provides little or no gain, and sometimes hurts, on Llama-3.1-8B math and on individualistic value reasoning, showing it depends on base model capability and confidence-correctness correlation.
Limitations
- EM is less effective when model confidence does not correlate with correctness, e.g. value alignment tasks such as IndVal.
- Gains are smaller for Llama-3.1-8B on math, suggesting dependence on base model reasoning capability.
- Entropy-only reward can reduce output diversity, diminishing benefits of combining EM-INF with self-consistency or refinement at larger compute.
Experimental setup
Datasets: NuminaMath Eurus-2-RL coding split IndVal MATH-500 AMC AIME 2024 Minerva Math OlympiadBench LeetCode LiveCodeBench v2 UGPhysics SciCode
Benchmarks: MATH-500 AMC AIME Minerva Math OlympiadBench LeetCode LiveCodeBench v2 UGPhysics SciCode IndVal
Metrics: Pass@1 accuracy FLOPs
Baselines: GRPO RLOO SFT Self-consistency RL Adaptive Temp Self-consistency Iterative-refinement GPT-4o Claude 3 Opus Gemini 1.5 Pro
Builds on: GRPO RLOO REINFORCE self-consistency adaptive temperature scaling
Closest papers by meaning
- Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization (NeurIPS 2025 · similarity 0.92)
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.91)
- Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy (ICLR 2026 · similarity 0.91)
- Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.90)
- Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Following (AAAI 2026 · similarity 0.89)
- The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.