Research map / Large language models / Reasoning and chain of thought

Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DARS, a difficulty-adaptive multi-stage rollout sampler that allocates extra rollouts to low-accuracy RLVR prompts, combined with full-batch large-breadth training (DARS-Breadth) to jointly improve Pass@1 and Pass@K on math reasoning.

Problem

GRPO-style RLVR under-weights hard, low-accuracy problems through a cumulative-advantage bias, capping Pass@K; increasing rollout size alone does not reliably help, and the role of training batch size (breadth) for Pass@1 is unclear.

Approach

Estimate per-prompt difficulty with a cheap pre-rollout, then allocate extra rollouts to hard prompts via ET or HW schedules that re-balance cumulative advantage (and induce log-odds or max-likelihood objectives); pair this with very large batches using full-batch PPO epochs so breadth boosts Pass@1 and sustains entropy.

Key results

• DARS-7B-HW-Breadth reaches Avg@128 of 58.4 vs 55.3 for the RLVR baseline on Qwen2.5-Math-7B, and Pass@128 of 83.4 vs 81.4.
• On Llama-3.1-8B, DARS-Llama-HW-Breadth reaches Pass@128 of 68.7 vs 56.5 for the GRPO baseline.
• DARS-ET cuts average rollouts per prompt by 52.5% for Qwen2.5-Math-1.5B relative to Depth-Naive (15.2 vs 32) while reducing time per step by 35.8%.

Measured effects reported in the paper

Surprising findings

• Naively increasing rollout size can harm Pass@K for Qwen2.5-Math-1.5B.
• Naively increasing batch size to 3072 harms Pass@128 for Qwen2.5-Math-1.5B even while improving Pass@1.

Limitations

• Experiments are limited to Qwen2.5-Math and Llama-3.1-8B base models and mainly to math reasoning.
• Full-batch updates with batch size 3072 are needed for breadth gains, which increases compute per step.
• Pass@K gains can decline with prolonged training, and checkpoint selection depends on test-set metrics.

Experimental setup

Datasets: OpenR1-45K MATH train split

Benchmarks: MATH500 OlympiadBench MinervaMath AIME24 AMC23 GPQA-Diamond HumanEval

Metrics: Pass@1 (Avg@128) Pass@128 Pass@K maj@8 maj@16

Baselines: Dr. GRPO (RLVR baseline) Depth-Naive (rollout size 32) Breadth-Naive (batch size 3072) MaxRL GRPO

Builds on: GRPO Dr. GRPO PPO LUFFY

Closest papers by meaning

• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.92)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.92)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.92)
• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.91)
• DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.91)
• What are Key Factors for Updates in RL for LLM Reasoning? (NeurIPS 2026 · similarity 0.91)