Research map / Reinforcement learning / RL from feedback

Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regime

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Scaling RL with verifiable rewards · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Prefix Sampling, a bidirectional replay of self-generated trajectory prefixes with prefix masking and adaptive per-bucket control that steers skewed binary-reward GRPO groups toward a 50% rollout pass rate, yielding faster convergence and wall-clock speedups on SWE-bench Verified and AIME 2025.

Problem

Binary-reward agentic RL for software engineering produces highly skewed, weakly contrastive rollout groups, wasting compute on uninformative rollouts under a fixed rollout budget.

Approach

Theoretically argues a 50% rollout pass rate maximizes reward entropy, filtering survival, RLOO advantage energy and pair count, then replays successful prefixes for hard groups and failing prefixes for easy groups (masking replayed tokens) with a per-bucket feedback controller to move rerollouts toward 50%.

Key results

• On SWE-bench Verified, Qwen3-14B reaches baseline peak at step 170 vs 300 with peak 0.295 (vs 0.274 baseline peak) and 2.01x end-to-end speedup.
• Qwen3-32B achieves 1.55x end-to-end speedup and 0.422 Pass@1 (+5.4 pp same-step gain over baseline at Ours peak).
• On AIME 2025, Qwen3-4B converges at step 110 vs 211 baseline (1.92x in ablation table) and Qwen3-8B reaches baseline peak at step 190 vs 260 with +10.8 pp same-step gain.

Surprising findings

• Prefix Sampling PS-fix without feedback control keeps the most valid groups per step yet converges later than the adaptive variant, indicating retained groups must land in the right operating regime.

Limitations

• Claims scoped to binary-reward RLVR with N=8 rollouts; 50% target is a reward-side signal target, not a global optimum.
• Math-reasoning step-time results are not diagnostic of replay-through-execution efficiency.
• Reported gains are same-step/average-over-8 evaluation comparisons, not independent training-seed estimates; some gaps within evaluation variability.

Experimental setup

Datasets: R2E-Gym-Subset AceReason-Math AceReason-Math-Subset

Benchmarks: SWE-bench Verified AIME 2025

Metrics: Pass@1 (avg over 8 runs) end-to-end wall-clock speedup eval/train steps to convergence step time valid groups per step

Baselines: DeepSWE/GRPO++ matched baseline PS-fix PS-ada hard-only

Builds on: GRPO DAPO dynamic sampling RLOO DeepSWE GRPO++ verl

Closest papers by meaning

• Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias (NeurIPS 2026 · similarity 0.87)
• SALT: When More Rollouts Don’t Help in Group-Based Policy Optimization and How to Make Them Matter (NeurIPS 2026 · similarity 0.87)
• PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization (NeurIPS 2026 · similarity 0.86)
• Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems (NeurIPS 2025 · similarity 0.86)
• Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
• Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.86)