AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes

Amrith Setlur, Zijian Wang, Andrew Cohen, Paria Rashidinejad, Sang Michael Xie

ICML 2026 · Reinforcement learning › RL from feedback · approach: Scaling RL with verifiable rewards · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Condition on off-policy prefixes (PrefixRL) instead of SFT+RLLlama-3.1-8B RL on hard problems · Training accuracy on no-prefix hard problemsaccuracy at matched compute: 45% greaterImproved
Condition on off-policy prefixes (PrefixRL) instead of baselinesQwen3-4B RL on hard problems · Training accuracy on no-prefix hard problemsaccuracy at matched compute: 30% greaterImproved
Use PrefixRL instead of SFT+RL including rejection-sampling costLlama-3.1-8B · hard-problem trainingcompute efficiency: roughly 2xImproved
Train with PrefixRL instead of standard RLLlama-3.1-8B · AIME'25pass@1: 38.2 to 61.3Improved
Train with PrefixRL instead of standard RLLlama-3.1-8B · HMMT'25pass@1: 29.2 to 49.4Improved
Train with PrefixRL instead of baselines, larger kLlama-3.1-8B · AIME'25pass@k gap: +18 points at k=8 and +28 at k=64Improved
Run PrefixRL with prefixes from Qwen3-4B-instruct instead of the Llama base model (ablation)Llama-3.1-8B PrefixRL · train and test math problemsaccuracyNo clear effect
Add a related in-context problem and its trace as prefix (PrefixRL on P2 given P3) vs RL on P2 alone (ablation)Llama-3.1-8B · In-context pair P2,P3pass@4 on P2: 18% to 63%Improved
Prefix with an unrelated in-context problem (ablation)Llama-3.1-8B PrefixRL · In-context pairs P3|P1, P1|P2pass@4No clear effect
Add SFT on off-policy traces before RL (ablation)Llama-3.1-8B RL · hard-problem trainingpass@64 and entropy: entropy as low as 0.01Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy