Research map / Continual learning / Continual fine-tuning of LLMs
Why Reinforcement Fine-Tuning Enables MLLMs Preserve Prior Knowledge Better: A Data Perspective
Zhihao Zhang, Qiaole Dong, Qi Zhang, Enyu Zhou, Jun Zhao, Zhiheng Xi, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Mingqi Wu, Yanwei Fu, Tao Ji and 3 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train SFT on GRPO-rollout reasoning data instead of human/GPT-4o data (ablation) | SFT · RefCOCO val, Qwen2.5-VL-7B | accuracy: 52.5 -> 81.4 | Improved |
| train SFT on GRPO-rollout reasoning data instead of GPT-4o reasoning data (ablation) | SFT · POPE, Qwen2.5-VL-3B | accuracy: 69.4 -> 85.4 | Improved |
| train SFT on GRPO-rollout data instead of GPT-4o data (ablation) | SFT · Jigsaw 3x3, Qwen2.5-VL-3B | accuracy: 70.0 -> 70.0 | No clear effect |
| use SFT with non-reasoning answers instead of RFT | SFT-Non-Rea · MME sum, Qwen2.5-VL-7B | score: 2333 -> 479 | Hurt |
| use RFT (GRPO) instead of SFT | RFT · prior benchmarks, Qwen2.5-VL-3B/7B | accuracy: RefCOCO val 88.8 -> 88.4 (3B), 90.0 -> 89.4 (7B) | Improved |
| add reasoning trajectories to SFT data (vs non-reasoning) (ablation) | SFT · prior knowledge, Qwen2.5-VL | LBK interference and forgetting: RefCOCO val 6.1 -> 74.2 (3B) | Improved |
| use SFT data of lower perplexity under base model (GRPO rollout vs 4o rollout) (ablation) | SFT · prior knowledge | forgetting: MME 7B 2084 -> 2207 | Improved |
| self-generate CoTs from one epoch of RFT then SFT (ablation) | SFT-Rea-Self-Generated · Jigsaw 3x3, Qwen2.5-VL-3B | accuracy: 70.0 -> 84.0 | Improved |
| train RFT/SFT on math reasoning with GRPO rollouts instead of Non-Rea (ablation) | SFT · GSM8k, Qwen2.5-7B-Instruct | accuracy: 21.8 -> 90.3 | Improved |
| raise SFT learning rate to 2e-5 with Rea-4o-Rollout (ablation) | SFT · RefCOCO val, Qwen2.5-VL-3B | accuracy: 88.8 -> 0.1 | Hurt |
| scale RFT to tens of thousands of steps | RFT · Jigsaw 3x3 | accuracy: 0.0 -> 66 (3B), 0.0 -> 75 (7B) | Improved |
| periodic restart of GRPO (reset optimizer and reference policy every 2 epochs) (ablation) | GRPO · Jigsaw training stability | KL divergence | Improved |
Closest papers by meaning
- Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning (NeurIPS 2026 · similarity 0.90)
- Understanding Catastrophic Forgetting in Language Models via Implicit Inference (ICLR 2024 · similarity 0.90)
- Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models (NeurIPS 2026 · similarity 0.89)
- Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem (ICML 2024 · similarity 0.89)
- UFT: Unifying Supervised and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.89)
- Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
