Research map / Large language models / Reasoning and chain of thought
Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DoPR, which selects one high-volatility, high-entropy training prompt per batch (EM-UCB acquisition over reward statistics) for GRPO updates, cutting rollout cost while matching GRPO on math benchmarks, and argues RLVR needs only ~16 training instances.
Problem
RLVR for reasoning LLMs is resource-intensive, requiring large reward-labeled datasets and many rollouts per batch (K×G per GRPO step).
Approach
Per batch, run one rollout per prompt to update EMA reward mean/variance and entropy, pick the single prompt maximizing reward volatility plus an entropy-gated UCB exploration bonus, and spend the full G-rollout GRPO budget only on that prompt.
Key results
- DoPR on Qwen 1.5B reaches 49.6 average pass@1, matching GRPO (49.6) while using one rollout-group per step instead of K×G rollouts.
- With a 10k rollout budget, DoPR (Qwen 1.5B) reaches 46.1 average, above GRPO at 50k rollouts (per the paper's claim), while GRPO at 10k gets 35.2.
- Under the fixed 5k-rollout budget, DoPR averages 42.2 vs 27.8 for GRPO and 37.5 for One-Shot RL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Single-sample selection per batch (DoPR) instead of full GRPO rollouts | Qwen2.5-Math-1.5B · Full benchmark suite, 1000 steps | Pass@1 avg: 49.6 vs 49.6 | No clear effect |
| Reduce training set to 16 samples (DoPR) (ablation) | DoPR on Qwen2.5-Math-1.5B · Six math benchmarks | Pass@1 avg: 49.6 at 16 samples vs 49.6 at full data | No clear effect |
| Reduce training set to 8 samples (ablation) | GRPO and DoPR on Qwen2.5-Math-1.5B · Six math benchmarks | Pass@1 avg: GRPO 48.4 to 44.0; DoPR 49.6 to 42.9 | Hurt |
| Reduce training set to 16 samples (GRPO) (ablation) | GRPO on Qwen2.5-Math-1.5B · Six math benchmarks | Pass@1 avg: 49.6 full data to 48.4 at 16 samples | Hurt |
| Single-sample DoPR vs GRPO under 10k total rollout budget | Qwen2.5-Math-1.5B · Six math benchmarks, 10k rollouts | Pass@1 avg: 46.1 vs 35.2 | Helped |
| Single-sample DoPR vs GRPO under 5k total rollout budget | Qwen2.5-Math-1.5B · Six math benchmarks, 5k rollouts | Pass@1 avg: 42.2 vs 27.8 | Helped |
| Single-sample DoPR vs GRPO at 250k rollout budget | Qwen2.5-Math-1.5B · Six math benchmarks, 250k rollouts | Pass@1 avg: 49.4 vs 49.6 | No clear effect |
| Single-sample One-Shot RL vs DoPR at 250k rollouts (One-Shot RL plateaus) | One-Shot RL on Qwen2.5-Math-1.5B · Six math benchmarks, 250k rollouts | Pass@1 avg: 43.6 vs 49.4 for DoPR | Hurt |
| Add EM-UCB exploration term (vs no exploration term) (ablation) | DoPR on Qwen2.5-Math-1.5B · Six math benchmarks, 10k rollout budget | Pass@1 avg: 45.7 vs 41.6 | Helped |
| Entropy-modulated UCB replaced by standard UCB (ablation) | DoPR on Qwen2.5-Math-1.5B · Six math benchmarks, 10k rollout budget | Pass@1 avg: 45.1 vs 45.7 | Hurt |
| Add EM-UCB exploration term (vs no exploration term) (ablation) | DoPR on Qwen2.5-Math-1.5B · Six math benchmarks, converged performance | Pass@1 avg: 49.6 vs 49.0 | No clear effect |
| Entropy-modulated UCB replaced by standard UCB (ablation) | DoPR on Qwen2.5-Math-1.5B · Six math benchmarks, converged performance | Pass@1 avg: 49.6 vs 49.5 | No clear effect |
Surprising findings
- Reports that GRPO trained on only 16 samples nearly matches full-data performance (48.4 vs 49.6 average), suggesting RLVR mainly activates existing capability.
- Ablation shows that after convergence, all sample-selection variants perform similarly, so selection mainly matters under tight rollout budgets.
Limitations
- Theoretical lower bound relies on strong assumptions (local smoothness, Polyak–Łojasiewicz condition) that are asserted rather than verified.
- Evaluated only on mathematical reasoning with Qwen and LLaMA base models; generalization to other domains is untested.
- Inconsistencies in the paper: GRPO-16 vs DoPR-16 comparisons and the claim that DoPR's rollout cost is G+(K-1) versus the single-rollout-per-prompt description differ, and reported baseline numbers conflict across tables.
Experimental setup
Datasets: DeepScaleR-Preview-Dataset MATH Minerva-MATH
Benchmarks: AIME24 AMC MATH Minerva Math OlympiadBench GSM8K
Metrics: pass@1 average accuracy
Baselines: GRPO One-Shot RL UFO
Builds on: GRPO RLVR One-Shot RL UFO UCB
Closest papers by meaning
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.93)
- Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.93)
- Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.92)
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.92)
- $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.92)
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.