Research map / Continual learning / Continual fine-tuning of LLMs
Modulating Merging Strengths via Joint Loss Estimation for LoRA-based Continual Learning
Kun Gu, De Cheng, Zhipeng Xu, Lingfeng He, Di Xu, Nannan Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a LoRA-based continual learning method that modulates the merging strength of each new task's LoRA update using a joint loss estimate, keeping a single inference model.
Problem
LoRA-based continual learning merging methods maintain a single inference model by scaling each newly learned LoRA update with one coefficient, which is hard to tune and can trade off plasticity and stability (abstract truncated; only the start is available).
Approach
Instead of a fixed or heuristic merging coefficient, estimate the joint loss of the merged model to set per-task merging strengths for LoRA updates, so that a single merged model balances old and new tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add HIM (element-wise Fisher-guided merging) to sequential LoRA fine-tuning (ablation) | LoRA continual learning · 10S-ImageNet-R | A_Last: 73.05 -> 80.36 | Improved |
| add TSP training to sequential LoRA fine-tuning (ablation) | LoRA continual learning · 10S-ImageNet-A | A_Last: 52.56 -> 56.57 | Improved |
| add TSP on top of HIM (ablation) | M2LE · 10S-ImageNet-R | A_Last: 80.36 -> 81.34 | Improved |
| add TSP on top of HIM (ablation) | M2LE · 10S-ImageNet-A | A_Last: 61.35 -> 62.45 | Improved |
| use HIM element-wise merging instead of a shared scalar EMA-style merge (ablation) | M2LE · 10S-ImageNet-A | A_Last: 59.64 -> 61.35 | Improved |
| use HIM instead of LoRA-P&M scalar merge (ablation) | M2LE · 10S-ImageNet-R | A_Last: 78.89 -> 80.36 | Improved |
| accumulate Fisher over all past tasks instead of only task t-1 (ablation) | HIM · 10S-ImageNet-R | A_Last: 76.23 -> 80.36 | Improved |
| replace TSP with SAM perturbation (ablation) | HIM · 10S-ImageNet-R | A_Last: 80.36 -> 79.55 | Hurt |
| use TSP instead of SAM (ablation) | HIM · 10S-ImageNet-A | A_Last: 60.46 -> 62.45 | Improved |
| add TSP to reduce forgetting (ablation) | HIM · 10S-ImageNet-A | Final forgetting: 14.01 -> 10.04 | Improved |
| recompute Fisher on old data instead of reusing accumulated Fisher (ablation) | HIM · 10S-ImageNet-R | A_Last: 80.67 -> 80.67 | No clear effect |
| increase LoRA rank from 32 to 64 (ablation) | M2LE · 20S-ImageNet-R | A_Last: 78.62 -> 78.20 | No clear effect |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- SD-LoRA — SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning (ICLR 2025) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Limitations
- Only the abstract opening was available; full method and results could not be verified
Experimental setup
Builds on: LoRA
Closest papers by meaning
- When the Merge Coefficient Stops Mattering: Proximity Regularized Merging for Continual LoRA Adaptation (NeurIPS 2026 · similarity 0.91)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.90)
- SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning (ICLR 2025 · similarity 0.89)
- BECAME: Bayesian Continual Learning with Adaptive Model Merging (ICML 2025 · similarity 0.89)
- Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models (NeurIPS 2025 · similarity 0.88)
- CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
