Research map / Reinforcement learning / RL from feedback
Proximal Supervised Fine-Tuning
Wenhong Zhu, Ruobing Xie, Rui Wang, Xingwu Sun, Di Wang, Pengfei Liu
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| clip the policy ratio (PSFT) instead of plain SFT | Qwen2.5-7B-Instruct · Out-of-domain average (7 benchmarks) | accuracy: 57.90 -> 61.26 | Improved |
| clip the policy ratio (PSFT) instead of plain SFT | Llama3.1-8B-Instruct · Out-of-domain average (7 benchmarks) | accuracy: 50.49 -> 59.25 | Improved |
| clip the policy ratio (PSFT) instead of plain SFT | Qwen2.5-7B-Instruct · IFEval loose | accuracy: 54.42 -> 73.03 | Improved |
| use PSFT instead of SFT, in-domain | Qwen2.5-7B-Instruct · In-domain math average | accuracy: 47.99 -> 46.98 | No clear effect |
| add a warm-up SFT phase before PSFT (ablation) | PSFT · In-domain math average | accuracy: 46.98 -> 48.17 (Qwen); 35.93 -> 38.36 (Llama) | Improved |
| add warm-up to PSFT (ablation) | PSFT · IFEval loose | accuracy: 73.03 -> 55.07 | Hurt |
| add KL regularization (coefficient 0.5) to SFT | SFT · In-domain and out-of-domain averages | accuracy: 57.90 -> 57.38 (OOD, Qwen) | No clear effect |
| use PSFT as cold start before GRPO instead of SFT | Qwen2.5-7B-Instruct · Out-of-domain average after RL | accuracy: 59.90 -> 64.06 | Improved |
| use PSFT as cold start before GRPO instead of SFT | Qwen2.5-7B-Instruct · In-domain math average after RL | accuracy: 52.40 -> 53.31 | Improved |
| use PSFT instead of SFT before DPO | Qwen3-4B-Base · AlpacaEval2 LC | LC win rate (%): 16.96 -> 23.29 | Improved |
| use PSFT instead of SFT on a vision-language model | Qwen2.5-7B-VL-Instruct · MMLU (text-only) | accuracy: 45.39 -> 62.44 | Improved |
| remove ratio clipping from PSFT (ablation) | PSFT · Qwen2.5-7B-Instruct SFT training | gradient stability and downstream results | Hurt |
Closest papers by meaning
- On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification (ICLR 2026 · similarity 0.90)
- Towards On-Policy SFT: Distribution Discriminant Theory and its Applications in LLM Training (ICML 2026 · similarity 0.89)
- PriFT: Prior-Support Guided Token Reweighting for Supervised Fine-Tuning (NeurIPS 2026 · similarity 0.89)
- Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections (NeurIPS 2025 · similarity 0.87)
- AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy (ICLR 2026 · similarity 0.86)
- Simple Policy Optimization (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
