Research map / Large language models / Reasoning and chain of thought

Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MENTOR, which injects expert-model tokens only at high-entropy decision points during RLVR rollouts (mixed-policy sampling with speculative acceleration plus a mixed-policy GRPO advantage) to balance effective and diverse exploration.

Problem

RLVR effectiveness depends on base model capability; existing expert-trajectory imitation methods improve effectiveness but reduce exploration diversity and accelerate entropy collapse.

Approach

Instead of imitating full expert trajectories, sample from a token-level mixture of policy and expert distributions with weight proportional to token entropy (only high-entropy decision points get expert guidance), accelerate this via modified speculative sampling, and train with a GRPO variant that gives expert-guided rollouts only positive excess advantage with an annealed weight.

Key results

• On Qwen2.5-7B-Base, MENTOR reaches 46.7 average vs 42.8 for on-policy RL and 44.1 for QuestA, with MATH 81.4 vs 76.8.
• On LLaMA3.1-8B-Base, MENTOR averages 23.8 vs 20.6 for on-policy RL; QuestA drops to 19.0 below on-policy RL.
• MENTOR achieves a 9.2% average gain in pass@32 over baselines, per the paper's diversity analysis.

Surprising findings

• QuestA, which uses expert hints, underperforms on-policy RL on LLaMA3.1-8B (19.0 vs 20.6 average).
• Random and perplexity-based expert guidance give little gain over GRPO, while entropy-based guidance gives large gains (MATH 81.4 vs 76.8).

Limitations

• Expert model must share vocabulary with the policy and requires extra forward passes, adding rollout cost.
• Experiments limited to math and a few out-of-domain benchmarks on 3B-8B base models with one RL setup.
• Some hyperparameters (alpha schedule, p=0.95 entropy quantile) are tuned without broad sensitivity analysis.

Experimental setup

Datasets: MATH OpenR1-Math-220K

Benchmarks: MATH AIME24 AIME25 AMC Minerva Olympiad GPQA ARC MMLU-Pro

Metrics: pass@1 avg@32 pass@32

Baselines: On-policy RL (GRPO) LUFFY QuestA Base

Builds on: GRPO DAPO LUFFY QuestA Speculative Sampling

Closest papers by meaning

• Diversity-Incentivized Exploration for Versatile Reasoning (ICLR 2026 · similarity 0.92)
• SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs (ICML 2026 · similarity 0.91)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.90)
• DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.89)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.89)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.89)