Research map / Reinforcement learning / RL from feedback
Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning
Dylan Zhang, Yufeng Xu, Haojin Wang, Qingzhi Chen, Hao Peng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace SFT with PEAR token-level weighting before GRPO | Qwen3-1.7B-Base · SynLogic | Pass@1 (%): 13.1 -> 38.3 | Improved |
| replace SFT with PEAR before GRPO | DeepSeek-Distill-Qwen-1.5B · AIME-25 | Pass@8 (%): 5 -> 35 | Improved |
| replace SFT with PEAR before GRPO | DeepSeek-Distill-Qwen-1.5B · Math average Pass@1 | accuracy (%): 14 -> 39 | Improved |
| replace SFT with PEAR before GRPO | Qwen3-8B-Base · AIME-25 Pass@8 | Pass@8 (%): 35 -> 35 | No clear effect |
| replace SFT with PEAR before GRPO | Qwen2.5-1.5B-Math · AMC23 Pass@8 | Pass@8 (%): 80 -> 78 | Hurt |
| use single-step token importance weight instead of SFT | Qwen3-1.7B-Base · Math average accuracy | accuracy (%): 23 -> 18 | Hurt |
| use suffix-based PEAR weight instead of single-step weight (ablation) | Qwen3-8B-Base · Math average accuracy | accuracy (%): 30 -> 41 | Improved |
| block size 8 instead of token-level (B=1) (ablation) | PEAR · SynLogic Qwen3-1.7B-Base | Pass@1 (%): 45.00 -> 68.75 | Improved |
| block size 4 instead of sequence-level weighting (ablation) | PEAR · SynLogic Qwen3-1.7B-Base | Pass@8 (%): 67.50 -> 83.75 | Improved |
| add negative examples to PEAR (ablation) | PEAR · SynLogic | Pass@1 | Improved |
| use Qwen3-32B proxy instead of true behavior policy (ablation) | PEAR black-box · SynLogic Qwen3-1.7B-Base | Pass@1 | No clear effect |
| widen clip range from [-0.08,0.08] to [-0.30,0.30] (ablation) | PEAR · SynLogic Qwen3-1.7B-Base | Pass@32 (%): 42.50 -> 51.25 | No clear effect |
Closest papers by meaning
- Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead (ICLR 2026 · similarity 0.91)
- SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning (NeurIPS 2026 · similarity 0.91)
- AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy (ICLR 2026 · similarity 0.91)
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.90)
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.89)
- The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
