Research map / Continual learning / Continual fine-tuning of LLMs
Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting
Howard Chen, Noam Razin, Karthik R Narasimhan, Danqi Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RL (GRPO) instead of SFT | post-training of instruct LMs · IFEval/MMLU/Countdown, Llama and Qwen | non-target tasks drop: Llama 8B MMLU drop 38.5 -> -0.2 | Improved |
| remove KL regularization from GRPO (beta 0.05 -> 0.0) (ablation) | GRPO · IFEval/MMLU/Countdown, four models | gain-drop tradeoff | No clear effect |
| remove KL regularization from GRPO (ablation) | GRPO · Llama models on IFEval | non-target tasks drop | Hurt |
| replace GRPO advantage estimate with plain REINFORCE (ablation) | GRPO · Llama 3.1 8B Instruct, IFEval | non-target tasks drop: 3.4 -> 7.7 | No clear effect |
| replace GRPO with REINFORCE (ablation) | GRPO · Llama 3.1 8B Instruct, Countdown | target gain: 60.4 -> 7.5 | Hurt |
| replace GRPO with REINFORCE (ablation) | GRPO · Qwen 2.5 7B Instruct, IFEval | target gain: 17.0 -> 5.7 | Hurt |
| regenerate SFT data at the start of each epoch (Iterative-SFT) instead of only from initial policy (ablation) | Self-SFT · Qwen 2.5 1.5B/7B, IFEval and MMLU | non-target drop relative to GRPO | Improved |
| use data generated only from the initial policy (Self-SFT) instead of expert data (ablation) | SFT · IFEval/MMLU/Countdown | non-target tasks drop | Mixed |
| lower SFT learning rate to 1e-5 (ablation) | Self-SFT · IFEval/MMLU/Countdown | gain and drop | Mixed |
| SFT on data generated during an RL run (ablation) | SFT · Appendix A.4.1 | non-target drop | Improved |
| start with a multi-modal rather than uni-modal training policy (ablation) | reverse KL vs forward KL simulation · Gaussian mixture simulation | non-target drop (overlap): 0.7 -> 0.03 | Improved |
Closest papers by meaning
- Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem (ICML 2024 · similarity 0.89)
- From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones (ICLR 2026 · similarity 0.89)
- PRISM: Demystifying Retention and Interaction in Mid-Training (ICML 2026 · similarity 0.89)
- Spurious Forgetting in Continual Learning of Language Models (ICLR 2025 · similarity 0.89)
- Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning (NeurIPS 2026 · similarity 0.88)
- Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
