Research map / Continual learning / Continual fine-tuning of LLMs
Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Perturb-and-Merge (P&M), a continual learning framework that merges each new task model into the previous model via a closed-form Fisher-based convex coefficient and uses task-vector-aligned stochastic perturbations during training, combined with LoRA.
Problem
Continual learning models use the most recent task's parameters for inference on all tasks, which makes them prone to catastrophic forgetting of earlier tasks.
Approach
After each task, infer with a convex merge of the previous model and the new task-specific model, with the merge coefficient derived in closed form from a diagonal empirical Fisher (Hessian proxy) to minimize total loss increase; during training, sample stochastic perturbations along the task vector (second-order finite-difference regularizer) at no extra forward/backward cost.
Key results
- LoRA-P&M reaches 79.95% Acc on ImageNet-R 10 tasks versus 65.72% for LoRA and 77.34% for SD-LoRA.
- On CIFAR100 10 tasks, LoRA-P&M reaches 88.45% Acc, above SD-LoRA's 88.01%.
- As a model merging method on LoRA, P&M outperforms CoFIMA by 3.86% on CUB-10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add post-training task-vector merging (P&M) to LoRA | LoRA · ImageNet-R 10 tasks | Acc: 65.72 -> 79.95 | Helped |
| add post-training task-vector merging (P&M) to LoRA | LoRA · ImageNet-A 10 tasks | Acc: 44.41 -> 56.57 | Helped |
| add post-training task-vector merging (P&M) to LoRA | LoRA · DN*-5 | Acc: 71.81 -> 84.71 | Helped |
| add post-training task-vector merging (P&M) to LoRA | LoRA · C100-10 | Acc: 72.58 -> 88.45 | Helped |
| add post-training task-vector merging (P&M) to LoRA | LoRA · CUB-10 | Acc: 64.82 -> 78.29 | Helped |
| add task-vector perturbation during training (ablation) | LoRA-M · INR-10 | Acc: 78.35 -> 79.95 | Helped |
| add task-vector perturbation during training (ablation) | LoRA-M · INR-20 | Acc: 74.26 -> 76.37 | Helped |
| add task-vector perturbation during training (ablation) | LoRA-M · INA-10 | Acc: 56.16 -> 56.57 | Helped |
| add task-vector perturbation during training (ablation) | LoRA-M · DN*-5 | Acc: 81.28 -> 84.71 | Helped |
| add task-vector perturbation during training (ablation) | LoRA-M · C100-10 | Acc: 86.57 -> 88.45 | Helped |
| add task-vector perturbation during training (ablation) | LoRA-M · CUB-10 | Acc: 74.98 -> 78.29 | Helped |
| use one global α instead of per-module α (ablation) | P&M · ImageNet-R 5 tasks | Acc: 80.53 -> 80.88 | No clear effect |
Surprising findings
- Per-module merging coefficients learned per LoRA module were highly similar, so a single global coefficient was used.
- Random Gaussian noise with matched Frobenius norm as perturbation hurt performance on some datasets (e.g., ImageNet-A dropped to 49.51 from 56.16 for LoRA-M).
Limitations
- Diagonal empirical Fisher may not capture the true loss curvature, so the merge coefficient is not guaranteed optimal.
- Online Fisher variant degrades on longer task sequences (20 tasks: 71.20 vs 76.37).
- Memory grows linearly with the number of tasks because one Fisher matrix is stored per task.
Experimental setup
Datasets: ImageNet-R ImageNet-A DomainNet CIFAR100 CUB200
Benchmarks: ImageNet-R ImageNet-A DomainNet CIFAR100 CUB200
Metrics: average accuracy average anytime accuracy
Baselines: L2P DualPrompt CODA-Prompt HiDe-Prompt InfLoRA SD-LoRA Model Averaging DARE CoMA CoFIMA Full Fine-Tuning LoRA
Builds on: LoRA Elastic Weight Consolidation task arithmetic SD-LoRA CoMA
Closest papers by meaning
- BECAME: Bayesian Continual Learning with Adaptive Model Merging (ICML 2025 · similarity 0.93)
- STAR: Stability-Inducing Weight Perturbation for Continual Learning (ICLR 2025 · similarity 0.92)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.90)
- MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging (NeurIPS 2025 · similarity 0.90)
- SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models (NeurIPS 2024 · similarity 0.90)
- Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.