Research map / Reinforcement learning / RL from feedback
Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes
Amrith Setlur, Zijian Wang, Andrew Cohen, Paria Rashidinejad, Sang Michael Xie
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Condition on off-policy prefixes (PrefixRL) instead of SFT+RL | Llama-3.1-8B RL on hard problems · Training accuracy on no-prefix hard problems | accuracy at matched compute: 45% greater | Improved |
| Condition on off-policy prefixes (PrefixRL) instead of baselines | Qwen3-4B RL on hard problems · Training accuracy on no-prefix hard problems | accuracy at matched compute: 30% greater | Improved |
| Use PrefixRL instead of SFT+RL including rejection-sampling cost | Llama-3.1-8B · hard-problem training | compute efficiency: roughly 2x | Improved |
| Train with PrefixRL instead of standard RL | Llama-3.1-8B · AIME'25 | pass@1: 38.2 to 61.3 | Improved |
| Train with PrefixRL instead of standard RL | Llama-3.1-8B · HMMT'25 | pass@1: 29.2 to 49.4 | Improved |
| Train with PrefixRL instead of baselines, larger k | Llama-3.1-8B · AIME'25 | pass@k gap: +18 points at k=8 and +28 at k=64 | Improved |
| Run PrefixRL with prefixes from Qwen3-4B-instruct instead of the Llama base model (ablation) | Llama-3.1-8B PrefixRL · train and test math problems | accuracy | No clear effect |
| Add a related in-context problem and its trace as prefix (PrefixRL on P2 given P3) vs RL on P2 alone (ablation) | Llama-3.1-8B · In-context pair P2,P3 | pass@4 on P2: 18% to 63% | Improved |
| Prefix with an unrelated in-context problem (ablation) | Llama-3.1-8B PrefixRL · In-context pairs P3|P1, P1|P2 | pass@4 | No clear effect |
| Add SFT on off-policy traces before RL (ablation) | Llama-3.1-8B RL · hard-problem training | pass@64 and entropy: entropy as low as 0.01 | Hurt |
Closest papers by meaning
- SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning (NeurIPS 2026 · similarity 0.89)
- Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.89)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.88)
- Nudging the Boundaries of LLM Reasoning (ICLR 2026 · similarity 0.88)
- Tricks or Traps? A Deep Dive into RL for LLM Reasoning (ICLR 2026 · similarity 0.88)
- Learning to Reason under Off-Policy Guidance (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
