Research map / Continual learning / Continual fine-tuning of LLMs
Model Sensitivity Aware Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MACL, a plug-in continual learning method that minimizes worst-case loss over a Gaussian parameter-distribution neighborhood (KL-constrained) using natural gradient in expectation-parameter space, reducing parameter sensitivity to balance forgetting and new-task accuracy.
Problem
Continual learning faces a trade-off between retaining old-task knowledge and learning new tasks well; existing methods often sacrifice one for the other.
Approach
Treats CL performance as dependent on model parameter sensitivity and optimizes the worst-case Gaussian parameter distribution within a KL neighborhood of the current distribution, using natural gradient descent in expectation-parameter space (avoiding explicit FIM) with a diagonal Hessian approximation; the method is a plug-in to existing CL methods.
Key results
- On Tiny-ImageNet Task-IL with 500 memory, DER++ improves from 51.91 to 54.03 when integrated with MACL.
- On CIFAR-100 Class-IL with 500 memory, DER++ improves from 72.70 to 74.53 on CIFAR-10 Class-IL, and CIFAR-100 Class-IL from 36.37 to 39.42 with MACL.
- Integration adds about 55% to 61% running time over the base CL method on CIFAR-100 per the paper's efficiency table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MACL to LODE | LODE · CIFAR-10 Class-IL | accuracy: 75.45 ± 0.90 -> 76.41 ± 0.67 | Helped |
| add MACL to LODE | LODE · CIFAR-10 Task-IL | accuracy: 94.41 ± 0.22 -> 94.32 ± 0.24 | Hurt |
| add MACL to LODE | LODE · CIFAR-100 Class-IL | accuracy: 38.95 ± 0.93 -> 40.67 ± 0.89 | Helped |
| add MACL to LODE | LODE · CIFAR-100 Task-IL | accuracy: 78.92 ± 0.67 -> 40.03 ± 0.51 | Hurt |
| add MACL to ER-ACE | ER-ACE · Tiny-ImageNet Class-IL | accuracy: 20.43 ± 0.97 -> 21.89 ± 0.83 | Helped |
| add MACL to LODE | LODE · Tiny-ImageNet Task-IL | accuracy: 60.18 ± 0.65 -> 61.79 ± 0.86 | Helped |
| add MACL to DER++ | DER++ · CIFAR-100 Task-IL | accuracy: 75.64 ± 0.60 -> 77.53 ± 0.89 | Helped |
| add MACL to DER++ | DER++ · CIFAR-100 Class-IL | accuracy: 36.37 ± 0.85 -> 39.42 ± 0.82 | Helped |
| add MACL to GPM | GPM · CIFAR-100 Task-IL | accuracy: 72.48 ± 0.40 -> 74.51 ± 0.36 | Helped |
| add MACL to DER++ | DER++ · CIFAR-10 Task-IL | accuracy: 93.88 ± 0.50 -> 94.72 ± 0.65 | Helped |
| add MACL to DER++ | DER++ · Tiny-ImageNet Class-IL | accuracy: 19.38 ± 1.41 -> 20.17 ± 1.56 | Helped |
| add MACL to DER++ | DER++ · Tiny-ImageNet Task-IL | accuracy: 51.91 ± 0.68 -> 54.03 ± 0.79 | Helped |
Surprising findings
- Overly reducing parameter sensitivity (large eta) degrades performance, so the benefit is non-monotonic in the regularization strength.
- Replacing natural gradient with plain gradient descent yields a smaller gain (DER++ MACL-GD 52.97 vs MACL-NGD 54.03 on Tiny-ImageNet Task-IL).
Limitations
- Introduces additional training cost compared to existing CL approaches.
- Hyperparameter eta must be tuned; excessive parameter-sensitivity reduction hurts new-task learning.
- Evaluated mainly on image classification benchmarks with ResNet18 and ViT backbones.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet 5-datasets CUB200 ImageNet-R
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet 5-datasets CUB200 ImageNet-R
Metrics: overall accuracy new task accuracy backward transfer
Baselines: EWC SI LwF NCL HAT GPM ER DER++ ER-ACE LODE
Builds on: DER++ SAM EWC UDIL CODA-Prompt
Closest papers by meaning
- BECAME: Bayesian Continual Learning with Adaptive Model Merging (ICML 2025 · similarity 0.91)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.91)
- A Unified and General Framework for Continual Learning (ICLR 2024 · similarity 0.91)
- PaLoRA: Paced Low-Rank Adaptation for Continual Learning (NeurIPS 2026 · similarity 0.90)
- SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models (NeurIPS 2024 · similarity 0.90)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.