Research map / Large language models / Reasoning and chain of thought
Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GRESO, an online pre-rollout filter that skips prompts predicted to be zero-variance (zero-advantage) in GRPO using their reward history, cutting rollout cost up to 2.4x while keeping accuracy comparable to Dynamic Sampling.
Problem
GRPO training wastes rollout compute on zero-variance prompts (all responses get identical rewards); Dynamic Sampling improves performance but needs many extra rollouts to refill batches with effective prompts.
Approach
Exploit temporal consistency of zero-variance status across epochs: each prompt's skip probability grows with its number of consecutive recent zero-variance epochs, with a self-adjusting base exploration probability and an adaptive rollout batch size, so uninformative prompts are filtered before rollout rather than after.
Key results
- On Qwen2.5-Math-7B (DM), GRESO matches DS average accuracy (57.5% vs 57.8%) while cutting rollouts from 13.1M to 6.3M.
- Rollout time reduced up to 2.4x (155.9h to 65.5h) and total training time up to 2.0x (178.0h to 88.3h) on Qwen2.5-Math-7B.
- Up to 3.35x fewer rollouts across the six settings with no notable accuracy drop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace post-rollout dynamic filtering with GRESO selective pre-rollout filtering | Qwen2.5-Math-1.5B trained on DM · Six-benchmark average | accuracy (%): 48.4 -> 48.5 | No clear effect |
| Replace post-rollout dynamic filtering with GRESO selective pre-rollout filtering | Qwen2.5-Math-1.5B trained on OR1 · Six-benchmark average | accuracy (%): 46.7 -> 46.8 | No clear effect |
| Replace post-rollout dynamic filtering with GRESO selective pre-rollout filtering | DeepSeek-R1-Distill-Qwen-1.5B trained on DM · Six-benchmark average | accuracy (%): 61.0 -> 60.5 | No clear effect |
| Replace post-rollout dynamic filtering with GRESO selective pre-rollout filtering | DeepSeek-R1-Distill-Qwen-1.5B trained on OR1 · Six-benchmark average | accuracy (%): 56.7 -> 57.3 | No clear effect |
| Replace post-rollout dynamic filtering with GRESO selective pre-rollout filtering | Qwen2.5-Math-7B trained on DM · Six-benchmark average | accuracy (%): 57.8 -> 57.5 | No clear effect |
| Replace post-rollout dynamic filtering with GRESO selective pre-rollout filtering | Qwen2.5-Math-7B trained on OR1 · Six-benchmark average | accuracy (%): 54.8 -> 55.1 | No clear effect |
| Use GRESO instead of Dynamic Sampling | Qwen2.5-Math-7B trained on DM · 1000 training steps | rollout time (hours): 155.9 hours to 65.5 hours; 2.4× speedup | Helped |
| Use GRESO instead of Dynamic Sampling | Qwen2.5-Math-7B trained on DM · 1000 training steps | total training time (hours): 178.0 to 88.3 hours; 2.0× speedup | Helped |
| Set target zero-variance percentage to 0% rather than 25% (ablation) | GRESO · Six-benchmark average | accuracy (%): 48.5 -> 48.1 | No clear effect |
| Set target zero-variance percentage to 50% rather than 25% (ablation) | GRESO · Six-benchmark average | accuracy (%): 48.5 -> 48.5 | No clear effect |
| Set target zero-variance percentage to 100% rather than 25% (ablation) | GRESO · Six-benchmark average | accuracy (%): 48.5 -> 48.4 | No clear effect |
| Add adaptive rollout batch size to Dynamic Sampling (ablation) | Dynamic Sampling · Qwen-Math-1.5B trained on DAPO + MATH | rollouts per training step | Helped |
Surprising findings
- About 20% of prompts previously labeled zero-variance become effective again in later epochs, so static pruning discards useful data.
- Linear backoff filtering causes periodic fluctuations in zero-variance ratio, which motivated probabilistic filtering.
Limitations
- Binary filtering only; does not rank or score the value of remaining prompts.
- Easy/hard allocation ratio (1:2) and target zero-variance percentage chosen heuristically; optimal allocation left open.
- Evaluated only on math reasoning with three models of 1.5B and 7B size.
Experimental setup
Datasets: DAPO MATH OPEN-R1 math 30k subset
Benchmarks: MATH500 AIME24 AMC Minerva Math Gaokao OlympiadBench
Metrics: avg@4 pass@1 accuracy number of rollouts wall-clock rollout time total training time
Baselines: Dynamic Sampling (DAPO) GRPO with Dynamic Sampling + ABS
Builds on: GRPO PPO DAPO verl
Closest papers by meaning
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.90)
- Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor’s Internal States (NeurIPS 2026 · similarity 0.90)
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale (NeurIPS 2025 · similarity 0.90)
- Nudging the Boundaries of LLM Reasoning (ICLR 2026 · similarity 0.89)
- Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration (ICML 2026 · similarity 0.89)
- CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.