Research map / Continual learning / Continual fine-tuning of LLMs

When the Merge Coefficient Stops Mattering: Proximity Regularized Merging for Continual LoRA Adaptation

NeurIPS 2026 · Continual learning › Continual fine-tuning of LLMs · approach: Low-rank and model-merging adaptation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PRM, which adds a proximal penalty on current-task LoRA factors during task-vector training so that sequential LoRA merges become less sensitive to the merge coefficient, improving AAA on rehearsal-free class-incremental benchmarks.

Problem

Rehearsal-free continual learning via sequential LoRA merging is fragile: performance depends sharply on the merge coefficient, and existing methods focus on choosing the coefficient rather than on making the task vector itself mergeable.

Approach

Add a proximal (L2-to-initialization) penalty on the current task's LoRA factors during task-vector training, leaving the downstream write-in rule unchanged; this shrinks the task-vector radius, lowers a norm-based bound on Fisher-weighted interference, and broadens the good coefficient plateau.</prompt_core>

Key results

• On ImageNet-R AugReg, PRM-fixed reaches AAA 86.39±0.53 at T=10 and 83.70±0.81 at T=20, best in those columns.
• Across seven AugReg settings, PRM-fixed has mean AAA 86.87 vs 86.38 for P&M using a fixed alpha=0.8.
• Ablation on ImageNet-R 10T: LoRA-Prox alone reaches AAA 85.61 and cuts Fisher interference from 45.12e-3 (P&M) to 4.93e-3.

Surprising findings

• Rescaling the P&M task vector to PRM's norm greatly reduces coefficient sensitivity, suggesting scale explains much of the sensitivity.
• NoWrite attribution shows much of the gain is present in the shaped task vector before write-in.

Limitations

• Empirical scope limited to rehearsal-free class-incremental vision benchmarks.
• Studied mainly under additive LoRA write-in; extension to other adapters or non-additive merges is open.
• Too strong proximal penalty harms current-task plasticity.

Experimental setup

Datasets: ImageNet-R CIFAR-100 DomainNet CUB-200

Benchmarks: ImageNet-R CIFAR-100 DomainNet CUB-200

Metrics: AAA final accuracy final forgetting plateau width Fisher-weighted interference

Baselines: LoRA EWC-LoRA Inf-LoRA SD-LoRA L2P CODA-Prompt Model-Avg MagMax CoFiMA CoMA BECAME P&M

Builds on: P&M BECAME CoFiMA MagMax LoRA

Closest papers by meaning

• Crowded in B-Space: Calibrating Shared Directions for LoRA Merging (NeurIPS 2026 · similarity 0.90)
• SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning (ICLR 2025 · similarity 0.89)
• CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning (CVPR 2025 · similarity 0.88)
• Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.88)
• IterIS: Iterative Inference-Solving Alignment for LoRA Merging (CVPR 2025 · similarity 0.88)
• Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.88)