AI Paper Atlas

Research map / Continual learning / Continual fine-tuning of LLMs

Modulating Merging Strengths via Joint Loss Estimation for LoRA-based Continual Learning

Kun Gu, De Cheng, Zhipeng Xu, Lingfeng He, Di Xu, Nannan Wang

NeurIPS 2026 · Continual learning › Continual fine-tuning of LLMs · approach: Low-rank and model-merging adaptation · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a LoRA-based continual learning method that modulates the merging strength of each new task's LoRA update using a joint loss estimate, keeping a single inference model.

Problem

LoRA-based continual learning merging methods maintain a single inference model by scaling each newly learned LoRA update with one coefficient, which is hard to tune and can trade off plasticity and stability (abstract truncated; only the start is available).

Approach

Instead of a fixed or heuristic merging coefficient, estimate the joint loss of the merged model to set per-task merging strengths for LoRA updates, so that a single merged model balances old and new tasks.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add HIM (element-wise Fisher-guided merging) to sequential LoRA fine-tuning (ablation)LoRA continual learning · 10S-ImageNet-RA_Last: 73.05 -> 80.36Improved
add TSP training to sequential LoRA fine-tuning (ablation)LoRA continual learning · 10S-ImageNet-AA_Last: 52.56 -> 56.57Improved
add TSP on top of HIM (ablation)M2LE · 10S-ImageNet-RA_Last: 80.36 -> 81.34Improved
add TSP on top of HIM (ablation)M2LE · 10S-ImageNet-AA_Last: 61.35 -> 62.45Improved
use HIM element-wise merging instead of a shared scalar EMA-style merge (ablation)M2LE · 10S-ImageNet-AA_Last: 59.64 -> 61.35Improved
use HIM instead of LoRA-P&M scalar merge (ablation)M2LE · 10S-ImageNet-RA_Last: 78.89 -> 80.36Improved
accumulate Fisher over all past tasks instead of only task t-1 (ablation)HIM · 10S-ImageNet-RA_Last: 76.23 -> 80.36Improved
replace TSP with SAM perturbation (ablation)HIM · 10S-ImageNet-RA_Last: 80.36 -> 79.55Hurt
use TSP instead of SAM (ablation)HIM · 10S-ImageNet-AA_Last: 60.46 -> 62.45Improved
add TSP to reduce forgetting (ablation)HIM · 10S-ImageNet-AFinal forgetting: 14.01 -> 10.04Improved
recompute Fisher on old data instead of reusing accumulated Fisher (ablation)HIM · 10S-ImageNet-RA_Last: 80.67 -> 80.67No clear effect
increase LoRA rank from 32 to 64 (ablation)M2LE · 20S-ImageNet-RA_Last: 78.62 -> 78.20No clear effect

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Limitations

Experimental setup

Builds on: LoRA

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy