Research map / Large language models / Reasoning and chain of thought

Reasoning with Exploration: An Entropy Perspective

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows entropy correlates with exploratory reasoning in LLMs (pivotal tokens, reflection, rare behaviors) and adds a clipped, gradient-detached entropy term to the advantage in GRPO/PPO to improve Pass@K on math benchmarks.

Problem

Standard RLVR training for LLM reasoning converges to narrow behaviors and plateaus, losing exploration; Pass@K gains vanish at large K.

Approach

Add a clipped, gradient-detached per-token entropy term to the advantage (A + min(alpha*H, |A|/kappa)), which amplifies updates at high-entropy exploratory tokens without adding an entropy gradient, unlike entropy regularization; a one-line code change.

Key results

• On AIME25 with Qwen2.5-Math-Base-7B, GRPO w/ entropy advantage reaches Pass@256 of 63.6 vs 57.4 for GRPO.
• On AIME25 with Qwen2.5-Base-7B, PPO w/ entropy advantage improves Pass@256 from 43.3 to 56.7 and Pass@1 from 7.9 to 11.7.
• On MATH500 with Qwen2.5-Base-7B, GRPO w/ entropy advantage raises Pass@16 from 65.4 to 74.0.

Surprising findings

• Adding an explicit entropy regularizer produced a sudden entropy spike after step 1500 and unstable optimization, while the entropy-based advantage stayed stable.
• Llama-series models abandoned intermediate reasoning chains within a few RL iterations, so RL was not successfully applied to them.

Limitations

• Experiments limited to 7B-scale Qwen models and one 1.5B code model; generality across model sizes is not established.
• Hyperparameters alpha and kappa are tuned per algorithm (alpha 0.4 for GRPO, 0.1 for PPO) with kappa fixed at 2.
• Pass@K gains on some benchmarks are small or negative (e.g., AIME24 with Qwen2.5-Math GRPO drops 3.3 points).

Experimental setup

Datasets: DAPO DeepCoder LiveCodeBench

Benchmarks: AIME 2025 AIME 2024 AMC 2023 MATH500 LiveCodeBench

Metrics: Pass@K Pass@1 repetition rate response length

Baselines: GRPO PPO RL with entropy regularization Qwen2.5-Math-Ins Eurus-2-PRIME Oat-Zero GPG

Builds on: PPO GRPO DAPO VAPO veRL

Closest papers by meaning

• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.93)
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.92)
• DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.91)
• Restoring Exploration after Post-Training: Latent Exploration Decoding for Large Reasoning Models (ICML 2026 · similarity 0.91)
• The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning (NeurIPS 2025 · similarity 0.90)
• Representation-Based Exploration for Language Models: From Test-Time to Post-Training (ICLR 2026 · similarity 0.90)