AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

PROS: Towards Compute-Efficient RLVR via Rollout Prefix Reuse

Baizhou Huang, Xiaojun Wan

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PROS, which reuses shared rollout prefixes across independent samples for the same query to cut RLVR rollout generation cost.

Problem

On-policy rollout generation in RLVR for large reasoning models is costly and grows with rollout length and model size, becoming the training bottleneck.

Approach

Empirically observes that independent rollouts for the same query share similar prefixes and exploits this by reusing rollout prefixes rather than generating each rollout from scratch.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add PROS (prefix reuse + Bayesian augmented-query selection)PPO · AIME24 (trained on DAPO-Train)Pass@1 (%): 28.23 -> 33.23Improved
add PROSPPO · AMC23 (trained on DAPO-Train)Pass@1 (%): 73.63 -> 78.20Improved
add PROSGRPO · AIME24 (trained on DAPO-Train)Pass@1 (%): 29.58 -> 34.17Improved
add PROSPPO · AIME24 and AMC23 (Qwen3-4B, AIME-Old)Pass@1 (%): AIME24 21.25 -> 27.40; AMC23 52.59 -> 62.12Improved
add Bayesian augmented-query selection on top of prefix reuse (ablation)PROS-ablation · AIME24 and AMC23 (PPO, DAPO-Train)Pass@1 (%): AIME24 31.35 -> 33.23; AMC23 73.93 -> 78.20Improved
add prefix reuse with random query sampling (PROS-ablation) (ablation)PPO · AIME24 (PPO, DAPO-Train)tokens per iteration (M): 10.24 -> 7.92Improved
replace PROS with Experience ReplayPPO/GRPO · AMC23 (DAPO-Train)Pass@1 (%): PPO 73.63 -> 73.02No clear effect
add Dynamic SamplingGRPO · AIME24 and AMC23 (AIME-Old)Pass@1 (%) and time: AIME24 31.04 -> 29.58; AMC23 59.98 -> 68.90; time 3.77 -> 45.38 minMixed
add Dynamic SamplingPPO · AMC23 (trained on DAPO-Train)Pass@1 (%): 73.63 -> 66.69Hurt
add Dynamic SamplingGRPO · per-iteration wall-clock (DAPO-Train)GPU minutes: 6.58 -> 19.45Hurt
add PROSGRPO · per-iteration wall-clock (DAPO-Train)GPU minutes: 6.58 -> 7.57Mixed
vary child-prior variance sigma and forgetting factor lambda (ablation)PROS (GRPO, AIME-Old) · AMC23 and AIME24Pass@1 (%): AMC23 62.42 to 67.53 across settings vs vanilla 59.98No clear effect

Experimental setup

Builds on: RLVR

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy