Research map / Large language models / Reasoning and chain of thought

Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes RL training dynamics of LLMs and VLMs, finding a two-phase shift from procedural consolidation to strategic planning exploration, and proposes HICRA, which amplifies advantages of planning tokens identified via strategic grams.

Problem

RL for LLM reasoning works, but its underlying learning dynamics (aha moments, length scaling, entropy dynamics) are poorly understood, and GRPO applies credit assignment uniformly across all tokens, diluting the learning signal for high-level strategic planning.

Approach

Decompose generated tokens into high-level planning tokens (identified as tokens within automatically mined Strategic Grams) and low-level execution tokens; show learning shifts from procedural consolidation to strategic planning exploration; then modify the GRPO advantage to amplify planning-token advantages by alpha*|A|.

Key results

• On Qwen3-4B-Instruct-2507, HICRA reaches 73.1 on AIME24 and 65.1 on AIME25 versus 68.5 and 60.0 for GRPO.
• On MiMO-VL-Instruct-2508, HICRA improves MathVision from 42.8 (GRPO) to 48.9 and MathVista from 73.7 to 80.7.
• Entropy regularization on Qwen2.5-7B-Base fails to consistently improve accuracy, while HICRA matches or beats GRPO on most benchmarks (e.g., AIME25 14.8 vs 11.4).

Surprising findings

• Token-level entropy can collapse while semantic entropy of strategic grams stays high and predicts validation accuracy.
• HICRA gives no advantage over GRPO on Llama-3.1-Instruct-8B, where the procedural foundation is lacking.

Limitations

• HICRA underperforms or ties GRPO when the base model lacks procedural reliability (Llama-3.1-Instruct-8B).
• Planning tokens are identified by a heuristic set of strategic grams mined from n-gram clustering, not an exhaustive or formally defined lexicon.
• Error-type analysis relies on GPT-4o classification of failures.

Experimental setup

Datasets: DAPO DeepScaleR ViRL39K

Benchmarks: AIME24 AIME25 Math500 AMC23 Minerva Olympiad MathVista MathVerse MathVision EMMA

Metrics: Pass@1 semantic entropy token entropy Pass@K

Baselines: GRPO Entropy Regularization ORZ SimpleRL Base model

Builds on: GRPO DAPO Clip-Higher entropy regularization

Closest papers by meaning

• Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning (ICML 2025 · similarity 0.90)
• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.90)
• Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models (ICML 2026 · similarity 0.89)
• Boosting LLM Reasoning via Human-Inspired Reward Shaping (NeurIPS 2026 · similarity 0.89)
• Plan Then Action: High-Level Planning Guidance Reinforcement Learning for LLM Reasoning (ICML 2026 · similarity 0.89)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.89)