Research map / Reinforcement learning / RL from feedback

ROLLVERIFY: BRIDGING EFFICIENCY AND ACCURACY IN LONG-TAIL ROLLOUT REINFORCEMENT LEARNING

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: GRPO stability and variants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RollVerify, which verifies and truncates off-policy suffixes of partial rollouts using an off-policy shift metric (OPS) before they enter training, matching on-policy GRPO accuracy at lower GPU cost.

Problem

Partial (asynchronous) rollout reduces GPU bubbles from long-tail rollouts in on-policy RL for LLMs, but stale off-policy samples degrade final accuracy, and existing fixes only reweight losses after generation.

Approach

Define a per-token and per-sequence off-policy shift metric (OPS) as the deviation of importance ratios from 1, then apply sequence-level and token-level verification to truncate invalid suffixes of unfinished rollouts before resuming them, plus a conditional switch back to on-policy rollout when acceptance rate drops.

Key results

• On Qwen3-8B-Base, RollVerify averages 57.2 vs 57.3 for on-policy GRPO while cutting training cost from 98.1 to 57.5 GPU days (1.7x).
• Naive partial rollout drops to 45.9 average accuracy on Qwen3-8B-Base despite 1.9x speedup.
• On Qwen3-30B-A3B-Base, RollVerify matches on-policy accuracy (62.5) at 65.1 vs 112 GPU days (1.7x).

Surprising findings

• Sequence-level verification alone raises accuracy but adding token-level verification also reduces GPU days (68.1 to 61.5), since outlier off-policy tokens destabilize training.
• Staleness-based filtering still yields entropy explosion and high OPS, suggesting staleness is a coarse proxy for off-policy deviation.

Limitations

• Evaluation focuses mainly on math; code and agentic results are preliminary and only in appendix.
• Only colocated setting tested; fully asynchronous disaggregated validation left to future work.
• No error bars or statistical significance tests reported.

Experimental setup

Datasets: DAPO-Math DeepScaleR ReTool DeepCoder

Benchmarks: AIME24 AIME25 AMC23 MATH500 LiveCodeBench HumanEval

Metrics: accuracy average accuracy GPU days OPS acceptance rate

Baselines: GRPO (on-policy) naive partial rollout staleness-based filtering GSPO SAPO VESPO

Builds on: GRPO partial rollout PPO speculative decoding decoupled clipping

Closest papers by meaning

• DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training (NeurIPS 2026 · similarity 0.90)
• Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction (NeurIPS 2026 · similarity 0.90)
• On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.89)
• EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts (NeurIPS 2026 · similarity 0.89)
• WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning (ICML 2026 · similarity 0.89)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.89)