Research map / Reinforcement learning / RL from feedback
On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, Jingren Zhou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace rigid SFT-then-RL switch with smoothly decaying SFT weight mu (CHORD-mu) | SFT+RL on Qwen2.5-7B · AMC | accuracy: 58.4 -> 60.8 | Improved |
| add token-wise weighting phi=p(1-p) on SFT loss (CHORD-phi) | GRPO pure RL · AIME25 | accuracy: 8.54 -> 17.2 | Improved |
| add CHORD-phi to pure RL | GRPO · BFCL Overall | accuracy: 77.1 -> 78.5 | Improved |
| decay mu dynamically instead of fixed mu (ablation) | CHORD-mu · reward curves | reward | Improved |
| minimal SFT before RL (SFT-light) vs original model | SFT-light · BFCL Overall | accuracy: 46.2 -> 34.0 | Hurt |
| start RL from SFT-light instead of SFT-best | SFT then RL · BFCL Overall vs AMC | accuracy: tool-use 76.1 -> 77.2; AMC 58.4 -> 52.5 | Mixed |
| use reward-adaptive mu (threshold 0.5 vs 0.7) (ablation) | CHORD-mu · BFCL Overall | accuracy: 78.1 (tau=0.5), 75.6 (tau=0.7) | Mixed |
| replace p(1-p) with focal-loss weight (ablation) | CHORD-phi · BFCL Overall | accuracy: 78.5 -> 73.4 | Hurt |
| replace p(1-p) with IS clipping (ablation) | CHORD-phi · BFCL Overall | accuracy: 78.5 -> 75.9 | Hurt |
| replace p(1-p) with entropy-based weights (ablation) | CHORD-phi · AIME2024 | accuracy: 14.0 -> 17.2 (Entropy Top); AMC23 59.7 -> 52.5 (Entropy Norm) | Mixed |
| apply CHORD-phi on weaker 3B policy | Qwen2.5-3B-Instruct · AMC23 | accuracy: 36.2 -> 41.9 vs SFT+RL | Improved |
| apply CHORD-phi to MoE model | Phi-mini-MoE-instruct · BFCL Overall | accuracy: 49.5 -> 61.6 | Improved |
Closest papers by meaning
- Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.91)
- Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.90)
- Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimization (NeurIPS 2025 · similarity 0.90)
- VESPO: Variational Sequence-level Soft Policy Optimization for Off-Policy LLM Training (NeurIPS 2026 · similarity 0.89)
- Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees (ICLR 2024 · similarity 0.89)
- Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
