Research map / Reinforcement learning / RL from feedback

SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Scaling RL with verifiable rewards · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that SFT baselines used by several mixed-policy RL papers were deflated by DeepSpeed CPU-offload and OpenRLHF loss-aggregation bugs, and that a corrected SFT-then-RL pipeline beats those methods on math benchmarks with Qwen2.5-Math-7B and Llama-3.1-8B.

Problem

Recent mixed-policy methods that blend SFT and RL for LLM reasoning report gains over SFT-then-RL, but their SFT baselines may be weakened by bugs in shared training frameworks, so the reported improvements may be inflated.

Approach

Identify two silent SFT bugs (a DeepSpeed CPU-offloaded optimizer dropping intermediate micro-batch gradients, and OpenRLHF's mean-of-per-mini-batch-means loss aggregation), fix them, and re-benchmark mixed-policy methods against a properly trained SFT->RL pipeline under controlled conditions.

Key results

• Corrected SFT->RL reaches 57.0 average on in-distribution math benchmarks with Qwen2.5-Math-7B, +3.8 over the next-best mixed-policy method SRFT (53.2).
• On Llama-3.1-8B, corrected SFT->RL averages 43.7 versus 21.5 for the best published mixed-policy method HPT, a +22.2 gap.
• Fixing only the CPU-offloaded optimizer bug raises the OpenRLHF SFT average from 48.3 to 53.4; fixing both reaches 54.0, matching verl's 53.8.

Surprising findings

• A 50-step truncated SFT->RL run outperforms LUFFY and ReLIFT on in-distribution math benchmarks while using fewer FLOPs.
• Reproducing a published SFT baseline with its own hyperparameters yielded a weaker score (48.3 vs 53.8), with the SFT learning rate and batch size being a major factor.

Limitations

• Limited to math reasoning with Qwen2.5-Math-7B and Llama-3.1-8B; no code or general-reasoning domains or larger scales.
• Mixed-policy methods were not tested atop a corrected SFT checkpoint, so a possible added value of mixed-policy stages is not assessed.
• ReLIFT and HPT SFT configurations could not be fully verified; LUFFY/ReLIFT reproductions are single-seed.

Experimental setup

Datasets: OpenR1-Math-46k-8192 OpenR1-Math-220k NuminaMath 1.5

Benchmarks: AIME24 AIME25 AMC MATH-500 Minerva OlympiadBench ARC-Challenge GPQA-Diamond MMLU-Pro

Metrics: avg@32 pass@1 average score

Baselines: LUFFY ReLIFT SRFT Prefix-RFT HPT

Builds on: GRPO DeepSpeed OpenRLHF verl LUFFY

Closest papers by meaning

• Tricks or Traps? A Deep Dive into RL for LLM Reasoning (ICLR 2026 · similarity 0.91)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.90)
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.90)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.89)
• Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.89)
• Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.89)