Research map / Large language models / Reasoning and chain of thought

Restoring Exploration after Post-Training: Latent Exploration Decoding for Large Reasoning Models

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that RL-post-trained reasoning models lose temperature-driven exploration (pass@n) because final-layer posteriors collapse, and proposes LED, a training-free decoding that samples from an entropy-maximizing aggregation of intermediate-layer posteriors during DeepThink.

Problem

RL post-trained large reasoning models show exploration collapse: raising sampling temperature no longer improves pass@n because the final-layer posterior is over-concentrated and low-entropy.

Approach

Intermediate-layer posteriors (decoded via the LM head, early-exit style) retain high entropy; LED top-k filters them onto final-layer candidates, aggregates them by a final-to-latent cumulative sum, and samples from the highest-entropy depth combination, gated by final-layer confidence and applied only in the DeepThink phase.

Key results

• LED improves average pass@1 by 0.61-0.67 and pass@16 by 0.92-1.03 points over CoT across benchmarks and models (e.g., Qwen3-4B-Thinking overall pass@1 78.20 to 79.32, pass@16 89.19 to 89.86).
• Generation length stays nearly unchanged (10,834 to 10,923 tokens on average).
• Using LED as the GRPO rollout policy yields 45.44% test accuracy under LED decoding versus 43.66% for a regular-rollout-trained model under LED.

Surprising findings

• Applying the final LayerNorm to latent states worsens pass@1 by 1.35 points while improving pass@16 by 0.79 points.
• Removing the top-k filtering causes endless looping, with most generations hitting the context limit.

Limitations

• Gains are modest (sub-point average improvements) and within noise for some benchmarks, with evaluation noted to possibly contain verifier false positives/negatives.
• Requires hyperparameters such as top-k=20 and exploration depth d, with d=8 chosen differently from the best d=12 on Qwen3-4B.
• Top-p is approximated by a smaller top-k rather than implemented directly.

Experimental setup

Datasets: GSM8K MATH-500 AIME2024 AIME2025 GPQA-Diamond LiveCodeBench v5

Benchmarks: AIME2024 AIME2025 GSM8K MATH-500 GPQA-Diamond LiveCodeBench v5

Metrics: pass@1 pass@16 generation length accuracy-temperature slope

Baselines: CoT DoLa SoftThinking SoftThinking-Gumbel

Builds on: DoLa SoftThinking GRPO top-k sampling

Closest papers by meaning

• Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.91)
• ReLaX: Reasoning with Latent Exploration for Large Reasoning Models (CVPR 2026 · similarity 0.90)
• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.88)
• Training Large Reasoning Models Efficiently via Progressive Thought Encoding (ICLR 2026 · similarity 0.88)
• Reasoning Palette: Modulating Reasoning via Latent Contextualization for Controllable Exploration for (V)LMs (CVPR 2026 · similarity 0.88)
• DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.88)