Research map / Continual learning / Continual fine-tuning of LLMs
Overcoming Multi-model Forgetting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Weight Plasticity Loss to ENAS search | ENAS · PTB | perplexity: 65.01 -> 61.9 | Helped |
| Add Weight Plasticity Loss to ENAS search | ENAS · CIFAR10 | top-1 error: 4.87% -> 3.81% | Helped |
| Add Weight Plasticity Loss while training the second model | Sequentially trained shared-parameter models · MNIST, strict convergence | forgetting reduction: up to 99% | Helped |
| Add Weight Plasticity Loss while training the second model | Sequentially trained shared-parameter models · MNIST, loose convergence | forgetting reduction: 52% | Helped |
| Use WPL with models trained to around 38% of optimal accuracy (ablation) | Sequentially trained shared-parameter models · MNIST, loose convergence | forgetting reduction: up to 2% | No clear effect |
| Add WPL during ENAS search | ENAS · PTB, sampled architectures | forgetting reduction: up to 95% | Helped |
| Add WPL during ENAS search | ENAS · PTB, five architectures with lowest initial error | forgetting reduction: 59% | Helped |
| Add WPL during ENAS search | ENAS · PTB, most affected sampled architecture | forgetting reduction: 51% | Helped |
| Add WPL to NAO with output dropout | NAO · PTB | forgetting | Helped |
| Use random path-dropping instead of WPL (ablation) | NAO · PTB | learning | Hurt |
| Vary output dropout rate from 0 to 0.75 (ablation) | NAO · PTB | forgetting | Mixed |
Closest papers by meaning
- Fortuitous Forgetting in Connectionist Networks (ICLR 2022 · similarity 0.88)
- On Plasticity, Invariance, and Mutually Frozen Weights in Sequential Task Learning (NeurIPS 2021 · similarity 0.88)
- Forget Sharpness: Perturbed Forgetting of Model Biases Within SAM Dynamics (ICML 2024 · similarity 0.87)
- Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective (ICML 2025 · similarity 0.86)
- On Local Overfitting and Forgetting in Deep Neural Networks (AAAI 2025 · similarity 0.86)
- Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.