Research map / Continual learning / Continual fine-tuning of LLMs

C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models

NeurIPS 2026 · Continual learning › Continual fine-tuning of LLMs · approach: Synthetic data and merging for CLIP

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes C-LoRA, which decomposes a LoRA routing matrix into a stop-gradient stability part and a gated plasticity part so a single fixed-size adapter learns sequential visual tasks without forgetting.

Problem

Single LoRA adapters forget rapidly in continual learning of pre-trained vision models, while existing fixes expand the adapter pool, increasing parameters and inference overhead.

Approach

Insert a learnable r×r routing matrix between LoRA factors (ΔW = A R B), split R into a frozen-gradient consolidated base and a trainable per-task delta, and gate the delta's gradients with an importance-based sigmoid mask so critical subspaces are protected within one shared adapter.

Key results

• On Split CUB-200 (10 sessions), C-LoRA reaches 89.93% Last-Acc versus 87.04% for Adam+adapter and 84.55% for SLCA.
• On Split CIFAR-100 (10 sessions), C-LoRA reaches 91.70% Last-Acc versus 87.53% for EASE; without stage-2 resampling it reaches 88.49%.
• Ablation on CUB-200 shows average forgetting reduced from 10.6 (no decomposition or gate) to 3.4 with full C-LoRA.

Surprising findings

• Adding a shared routing matrix without decomposition improves over vanilla LoRA by at most 1.91 Last-Acc points, suggesting extra capacity alone does not explain the gains.
• On CUB-200 with 20 sessions, C-LoRA's Last-Acc slightly increases with the number of sessions, attributed to overlap with pre-training categories.

Limitations

• Theoretical guarantees rely on simplified assumptions (bounded norms, i.i.d. zero-mean routing elements) that do not hold exactly in training.
• Experiments cover only class-incremental image classification with ViT backbones; detection, segmentation and other backbones are untested.
• Ablations use a single random seed.

Experimental setup

Datasets: CIFAR-100 ImageNet-A CUB-200 CAR196 ImageNet-R ImageNet-21K

Benchmarks: Split CIFAR-100 Split ImageNet-A Split CUB-200 Split CAR196 Split ImageNet-R

Metrics: Last-Acc Inc-Acc BWT forgetting

Baselines: EWC L2P DualPrompt SLCA Adam+adapter EASE InfLoRA SD-LoRA AdaLoRA CL-LoRA

Builds on: LoRA AdaLoRA SLCA Vision Transformer

Closest papers by meaning

• PaLoRA: Paced Low-Rank Adaptation for Continual Learning (NeurIPS 2026 · similarity 0.92)
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.92)
• SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning (ICLR 2025 · similarity 0.91)
• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.90)
• LoRAtorio: An intrinsic approach to LoRA Skill Composition (NeurIPS 2026 · similarity 0.90)
• CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning (CVPR 2025 · similarity 0.90)