Research map / Reinforcement learning / RL from feedback

Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: GRPO stability and variants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FastRL, a plug-and-play GRPO-variant framework that prunes redundant rollouts via advantage-grouped Jaccard-based diversity pruning (MDAP) and adapts rollout count using historical pruning statistics (ARS), achieving ~2x training speedup with small accuracy gains on visual math reasoning.

Problem

GRPO-style methods require multi-rollout sampling and per-token probability evaluation for every trajectory, which is computationally expensive, and many trajectories are low-information or homogeneous, weakening the learning signal.

Approach

Within each advantage-value group of a prompt's rollouts, greedily keep only trajectories whose token n-gram Jaccard distance to already-kept ones exceeds a threshold (MDAP), and adjust the next epoch's rollout count based on the retained-update statistics (ARS) so that redundant rollouts are sampled less.

Key results

• On Geometry3K with Qwen2.5-VL-7B-Instruct, FastRL reaches 2.14x training speedup over GRPO and average out-of-domain accuracy 52.63 vs 51.32 for GRPO.
• On GeoQA8K-R1V with Qwen2.5-VL-7B, FastRL achieves 2.22x speedup with GSPO and 2.13x with GRPO, improving average accuracy by 1.09 points.
• Removing MDAP drops speed to 1.15x on Geometry3K, showing MDAP is the main source of acceleration; CPPO achieves 1.60x but lower accuracy (51.68).

Surprising findings

• Advantage-only pruning (one trajectory per advantage group, no Jaccard pruning) reaches a higher speedup (2.35x) than full FastRL (2.14x) on Geometry3K, though with lower accuracy.
• Random pruning of 25-75% of trajectories degrades accuracy only modestly at low ratios but speeds up training less reliably than the advantage-based approach.

Limitations

• MDAP is designed for discrete reward signals; extending to continuous rewards requires modifications such as discretization.
• Speedup-accuracy trade-off is governed by threshold tau; higher tau speeds training but reduces accuracy.
• Experiments are limited to a few datasets and small-to-medium models, with single-run reporting of results.

Experimental setup

Datasets: Geometry3K GeoQA8K-R1V MATH OS-Atlas

Benchmarks: MathVerse MathVision MathVista WeMath AIME2023 AIME2024 AIME2025 AIME2026

Metrics: accuracy avg@16 pass@16 training time speedup

Baselines: GRPO DAPO GSPO CPPO GRESO

Builds on: GRPO DAPO GSPO CPPO GRESO EasyR1

Closest papers by meaning

• Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization (ICML 2026 · similarity 0.91)
• TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback (NeurIPS 2026 · similarity 0.89)
• SALT: When More Rollouts Don’t Help in Group-Based Policy Optimization and How to Make Them Matter (NeurIPS 2026 · similarity 0.88)
• Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation (NeurIPS 2026 · similarity 0.88)
• Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts (NeurIPS 2025 · similarity 0.87)
• Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.87)