Research map / Reinforcement learning / RL from feedback
Learning to Reason under Off-Policy Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LUFFY, which augments RLVR with off-policy reasoning traces from DeepSeek-R1 via Mixed-Policy GRPO and regularized importance sampling policy shaping, improving math and OOD reasoning over on-policy RL.
Problem
On-policy RLVR is bounded by the base model's capabilities and suffers entropy collapse, so it cannot acquire reasoning behaviors beyond its initial abilities and fails to train weak models.
Approach
Mixes off-policy teacher traces (DeepSeek-R1) into the GRPO group for advantage computation with an off-policy importance-weighted objective (pi_phi=1, no clipping), and shapes the importance weight with f(x)=x/(x+gamma) to up-weight low-probability tokens and preserve entropy.
Key results
- LUFFY reaches 50.1 average on six math benchmarks with Qwen2.5-Math-7B, +6.4 over prior RLVR methods and +4.6 over on-policy RL.
- On out-of-distribution benchmarks LUFFY averages 57.8, +6.2 over the best RLVR baseline OpenReasoner-Zero.
- On LLaMA-3.1-8B, on-policy RL collapses on the Hard set while LUFFY shows stable reward improvement; LUFFY improves over on-policy RL by +3.6 on LLaMA-3.1-8B average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add off-policy guidance with policy shaping and NoClip to On-Policy RL | Qwen2.5-Math-7B · In-distribution math benchmarks | average score: 45.5 -> 50.1 | Helped |
| add off-policy guidance with policy shaping and NoClip | Qwen2.5-Math-7B · Out-of-distribution benchmarks | average score: +6.2 points | Helped |
| add policy shaping to Mixed-Policy RL (ablation) | Mixed-Policy RL · In-distribution math benchmarks | average score: 44.4 -> 47.8 | Helped |
| remove on-policy clipping from Mixed-Policy RL with shaping (ablation) | Mixed-Policy RL + Shaping · In-distribution math benchmarks | average score: 47.8 -> 50.1 | Helped |
| add policy shaping without off-policy guidance (ablation) | On-Policy RL · In-distribution math benchmarks | average score: 45.5 -> 42.0 | Hurt |
| remove on-policy clipping without off-policy guidance (ablation) | On-Policy RL · In-distribution math benchmarks | average score: 45.5 -> 44.9 | Hurt |
| train LUFFY for extra steps | LUFFY · Out-of-distribution benchmarks | average score: 57.8 -> 61.8 | Helped |
| use policy shaping with γ=0.1 rather than other tested γ values (ablation) | LUFFY · Qwen2.5-Math-7B math benchmarks | accuracy: 50.1 | Helped |
Surprising findings
- Naively combining SFT with RL (SFT+RL, RL w/ SFT Loss) produces excessively long generations and higher compute cost than LUFFY.
- Applying shaping or no-clip to on-policy RL without off-policy guidance does not improve performance and can hurt it (e.g., On-Policy + Shaping drops to 42.0).
Limitations
- Only math tasks with verifiable golden answers are studied.
- Experiments limited to 7B and smaller models due to compute.
- Uses only one off-policy trajectory per question from a single teacher.
Experimental setup
Datasets: OpenR1-Math-220k NuminaMath 1.5
Benchmarks: AIME 2024 AIME 2025 AMC MATH-500 Minerva OlympiadBench ARC-c GPQA-diamond MMLU-Pro
Metrics: avg@32 accuracy pass@1 accuracy
Baselines: SimpleRL-Zero OpenReasoner-Zero PRIME-Zero Oat-Zero On-Policy RL (Dr.GRPO) SFT RL w/ SFT Loss SFT+RL
Builds on: GRPO PPO Dr.GRPO DeepSeek-R1 distillation
Closest papers by meaning
- Boosting Reasoning in Large Multimodal Models via Activation Replay (CVPR 2026 · similarity 0.89)
- Reason in the Words You Speak: Idiolectal Paraphrasing Off-Policy Traces for Reasoning Distillation in VideoLLMs (NeurIPS 2026 · similarity 0.89)
- Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners (ICLR 2026 · similarity 0.89)
- Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.89)
- The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.88)
- TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.