Research map / Continual learning / Continual fine-tuning of LLMs

SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning

ICLR 2025 · Continual learning › Continual fine-tuning of LLMs · approach: Low-rank and model-merging adaptation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SD-LoRA, a rehearsal-free, inference-efficient, end-to-end trainable LoRA method for class-incremental learning that decouples LoRA magnitude (learnable scalars per task) from direction (learned per task, previous directions frozen), with rank-reduction and least-squares distillation variants.

Problem

Class-incremental learning with pre-trained foundation models (ViTs) suffers catastrophic forgetting, and existing prompt-pool or LoRA-pool methods either grow components with task count, rely on task-specific selection at inference, or store replay samples, hurting scalability.

Approach

Decouple LoRA update into learnable magnitudes and normalized directions; freeze previously learned directions and only learn their scalar magnitudes plus the new direction, so the model is evaluated directly without component selection, and the learned magnitudes reweight earlier directions to reach a low-loss region shared across tasks.

Key results

• On ImageNet-R (N=20), SD-LoRA reaches 75.26% Acc, surpassing InfLoRA (69.89%) by 7.68 points as reported in the text.
• On ImageNet-A (N=10), SD-LoRA reaches 55.96% Acc and 64.95% AAA versus HiDe-Prompt's 42.70% Acc and 56.32% AAA.
• SD-LoRA-RR and SD-LoRA-KD show only marginal drops from full SD-LoRA (e.g., 74.05% and 74.12% Acc on ImageNet-R N=20).

Measured effects reported in the paper

Surprising findings

• Fixing a single learned LoRA direction and only learning its magnitude already outperforms vanilla LoRA in continual learning, suggesting early directions are critical.
• Learned magnitudes for earlier tasks rise while those for later tasks decline, so later directions act as minor refinements.

Limitations

• Evaluated only on ViT-based foundation models; extension to other architectures is left to future work.
• SD-LoRA-KD relies on a hand-set residual threshold and SD-LoRA-RR on predefined rank-reduction task indices.
• Theoretical analysis relies on assumptions on near-optimal update neighborhoods and simplified matrix factorization setting.

Experimental setup

Datasets: ImageNet-R ImageNet-A DomainNet CIFAR-100 CUB-200

Benchmarks: ImageNet-R (N=5/10/20) ImageNet-A (N=10) DomainNet (N=5) CIFAR-100 (N=10) CUB-200 (N=10)

Metrics: Acc AAA

Baselines: L2P DualPrompt CODA-Prompt HiDe-Prompt InfLoRA Full Fine-Tuning

Builds on: LoRA InfLoRA CODA-Prompt DoRA

Closest papers by meaning

• CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning (CVPR 2025 · similarity 0.93)
• C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models (NeurIPS 2026 · similarity 0.91)
• LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning (ICLR 2025 · similarity 0.90)
• LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learning (CVPR 2025 · similarity 0.90)
• Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models (NeurIPS 2025 · similarity 0.89)
• When the Merge Coefficient Stops Mattering: Proximity Regularized Merging for Continual LoRA Adaptation (NeurIPS 2026 · similarity 0.89)