Research map / Continual learning / Continual fine-tuning of LLMs
Self-Distillation Enables Continual Learning
Idan Shenfeld, Mehul Damani, Jonas Hübotter, Pulkit Agrawal
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use on-policy self-distillation (SDFT) instead of SFT | Qwen2.5-7B-Instruct · Science Q&A | new-task accuracy (%): 66.2 -> 70.2 | Improved |
| use SDFT instead of SFT | Qwen2.5-7B-Instruct · Science Q&A | prior-task average (%): 53.4 -> 64.5 | Improved |
| use SDFT instead of SFT | Qwen2.5-7B-Instruct · Tool Use | prior-task average (%): 56.0 -> 65.4 | Improved |
| use SDFT instead of SFT | Qwen2.5-7B-Instruct · Medical | new-task accuracy (%): 35.5 -> 40.2 | Improved |
| add re-invocation after SFT | SFT · Science Q&A | prior-task average (%): 53.4 -> 60.2 | Improved |
| use SDFT instead of SFT | Qwen2.5-7B-Instruct · Knowledge acquisition | OOD accuracy (%): 80 -> 98 | Improved |
| use SDFT instead of SFT | Qwen2.5-7B-Instruct · Knowledge acquisition | strict accuracy (%): 80 -> 89 | Improved |
| use SDFT instead of SFT on answer-only data | Olmo-3-7B-Think · Medical | accuracy (%): 23.5 -> 43.7 | Improved |
| condition the teacher on article text plus answer rather than text only (ablation) | SDFT · Knowledge acquisition | strict accuracy (%): 75 -> 89 | Improved |
| train on-policy instead of offline distillation from the same teacher (ablation) | SDFT · Tool Use | accuracy | Improved |
| increase model size from 3B to 14B (ablation) | SDFT vs SFT · Science Q&A | gap over SFT: four-point (7B) to seven points (14B) | Improved |
| use EMA teacher instead of frozen base or current student teacher (ablation) | SDFT · training stability | training stability | Improved |
Closest papers by meaning
- Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning (NeurIPS 2026 · similarity 0.89)
- Learning from Language Feedback via Variational Policy Distillation (NeurIPS 2026 · similarity 0.88)
- Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (NeurIPS 2026 · similarity 0.88)
- Self-Composing Policies for Scalable Continual Reinforcement Learning (ICML 2024 · similarity 0.87)
- Active Fine-Tuning of Multi-Task Policies (ICML 2025 · similarity 0.87)
- Self-Improving Embodied Foundation Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
