Research map / Large language models / Reasoning and chain of thought
PROS: Towards Compute-Efficient RLVR via Rollout Prefix Reuse
Baizhou Huang, Xiaojun Wan
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PROS, which reuses shared rollout prefixes across independent samples for the same query to cut RLVR rollout generation cost.
Problem
On-policy rollout generation in RLVR for large reasoning models is costly and grows with rollout length and model size, becoming the training bottleneck.
Approach
Empirically observes that independent rollouts for the same query share similar prefixes and exploits this by reusing rollout prefixes rather than generating each rollout from scratch.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add PROS (prefix reuse + Bayesian augmented-query selection) | PPO · AIME24 (trained on DAPO-Train) | Pass@1 (%): 28.23 -> 33.23 | Improved |
| add PROS | PPO · AMC23 (trained on DAPO-Train) | Pass@1 (%): 73.63 -> 78.20 | Improved |
| add PROS | GRPO · AIME24 (trained on DAPO-Train) | Pass@1 (%): 29.58 -> 34.17 | Improved |
| add PROS | PPO · AIME24 and AMC23 (Qwen3-4B, AIME-Old) | Pass@1 (%): AIME24 21.25 -> 27.40; AMC23 52.59 -> 62.12 | Improved |
| add Bayesian augmented-query selection on top of prefix reuse (ablation) | PROS-ablation · AIME24 and AMC23 (PPO, DAPO-Train) | Pass@1 (%): AIME24 31.35 -> 33.23; AMC23 73.93 -> 78.20 | Improved |
| add prefix reuse with random query sampling (PROS-ablation) (ablation) | PPO · AIME24 (PPO, DAPO-Train) | tokens per iteration (M): 10.24 -> 7.92 | Improved |
| replace PROS with Experience Replay | PPO/GRPO · AMC23 (DAPO-Train) | Pass@1 (%): PPO 73.63 -> 73.02 | No clear effect |
| add Dynamic Sampling | GRPO · AIME24 and AMC23 (AIME-Old) | Pass@1 (%) and time: AIME24 31.04 -> 29.58; AMC23 59.98 -> 68.90; time 3.77 -> 45.38 min | Mixed |
| add Dynamic Sampling | PPO · AMC23 (trained on DAPO-Train) | Pass@1 (%): 73.63 -> 66.69 | Hurt |
| add Dynamic Sampling | GRPO · per-iteration wall-clock (DAPO-Train) | GPU minutes: 6.58 -> 19.45 | Hurt |
| add PROS | GRPO · per-iteration wall-clock (DAPO-Train) | GPU minutes: 6.58 -> 7.57 | Mixed |
| vary child-prior variance sigma and forgetting factor lambda (ablation) | PROS (GRPO, AIME-Old) · AMC23 and AIME24 | Pass@1 (%): AMC23 62.42 to 67.53 across settings vs vanilla 59.98 | No clear effect |
Experimental setup
Builds on: RLVR
Closest papers by meaning
- Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration (ICML 2026 · similarity 0.92)
- Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.91)
- On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.91)
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.91)
- Generalization of RLVR Using Causal Reasoning as a Testbed (ICLR 2026 · similarity 0.90)
- Stable and Efficient Single-Rollout RL for Multimodal Reasoning (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
