Research map / Continual learning / Continual fine-tuning of LLMs
LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRA Subtraction (LoRA−), which negates old-task LoRA weights to build a drift-resistant gradient-projection space plus an augmented triplet loss, for exemplar-free class-incremental learning on a frozen ViT.
Problem
Catastrophic forgetting from feature drift in exemplar-free continual (class-incremental) learning, where old-task samples cannot be stored and existing subspace methods rely on static, outdated statistics for long task sequences.
Approach
Subtract the layer-wise LoRA weights of previous tasks from the pre-trained weights before training the new task, compute the input covariance under this modified model, and project gradients onto its top principal components (drift-resistant space); add an augmented triplet loss using old-class prototypes to restore plasticity.
Key results
- On ImageNet-R with 50 tasks, LoRA−DRS reaches 77.93% average accuracy versus 75.35% for EASE (runner-up in Tab. 1), a margin of about 2.6 points.
- On CIFAR-100 with 50 tasks, LoRA−DRS achieves 91.29% ACC_50 versus 87.65% for EASE.
- Ablation on ImageNet-R at 50 tasks: removing DRS drops ACC by 17.47 points; adding ATL improves ACC by 1.2 to 1.9 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LoRA Subtraction to construct DRS instead of training with LoRA alone (ablation) | LORA−DRS · ImageNet-R, 50 tasks | average accuracy: 58.56 -> 76.03 | Helped |
| add DRS to LoRA training (ablation) | LORA−DRS · ImageNet-R, 25 tasks | average accuracy: 67.13 -> 78.00 | Helped |
| add DRS to LoRA training (ablation) | LORA−DRS · ImageNet-R, 10 tasks | average accuracy: 76.03 -> 80.32 | Helped |
| add ATL to DRS training (ablation) | LORA−DRS · ImageNet-R, 10 tasks | average accuracy: 80.32 -> 81.52 | Helped |
| add ATL to DRS training (ablation) | LORA−DRS · ImageNet-R, 25 tasks | average accuracy: 78.00 -> 79.79 | Helped |
| add ATL to DRS training (ablation) | LORA−DRS · ImageNet-R, 50 tasks | average accuracy: 76.03 -> 77.93 | Helped |
| add ATL to DRS training (ablation) | LORA−DRS · ImageNet-R, 50 tasks | new class accuracy: 75.83 -> 77.88 | Helped |
| add ATL (ablation) | InfLoRA · ImageNet-R, 50 tasks | average accuracy: 66.27 -> 61.46 | Hurt |
| use LORA−DRS instead of EASE | ImageNet-R · 50 tasks | average accuracy: 75.77±0.58 -> 77.94±0.74 | Helped |
| use LORA−DRS instead of EASE | CIFAR-100 · 50 tasks | average accuracy: 87.65±0.46 -> 91.29±0.29 | Helped |
Surprising findings
- Adding the augmented triplet loss to InfLoRA lowers its ImageNet-R 50-task accuracy by 4.81 points, showing plasticity gains depend on controlling drift.
Limitations
- LoRA adapter modules still add model size overhead relative to the frozen backbone.
- Gradient projection can impair plasticity, which the triplet loss only partially offsets.
- Gains from the triplet loss depend on drift being controlled; it hurts InfLoRA on long sequences.
Experimental setup
Datasets: ImageNet-R CIFAR-100
Benchmarks: ImageNet-R CIFAR-100
Metrics: average accuracy final accuracy backward transfer new-class accuracy
Baselines: LoRA-FT LAE L2P DualPrompt CODA-Prompt InfLoRA Adam-NSCL EASE
Builds on: LoRA InfLoRA Adam-NSCL task arithmetic EASE
Closest papers by meaning
- LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning (ICLR 2025 · similarity 0.90)
- PaLoRA: Paced Low-Rank Adaptation for Continual Learning (NeurIPS 2026 · similarity 0.90)
- SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning (ICLR 2025 · similarity 0.90)
- Resurrecting Old Classes with New Data for Exemplar-Free Continual Learning (CVPR 2024 · similarity 0.89)
- C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models (NeurIPS 2026 · similarity 0.89)
- An Effective Dynamic Gradient Calibration Method for Continual Learning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.