Research map / Continual learning / Continual fine-tuning of LLMs
CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CL-LoRA, a dual-adapter LoRA framework for rehearsal-free class-incremental learning on a frozen ViT-B/16, combining task-shared adapters (fixed random orthogonal down-projection, early-exit distillation, gradient reassignment) with task-specific adapters using learnable block-wise weights.
Problem
Adapter-based PTM methods for rehearsal-free class-incremental learning create new adapters per task, causing parameter redundancy and failing to share knowledge across tasks, while continually updating shared modules causes forgetting.
Approach
Split the ViT into early blocks with a single continually updated shared LoRA (fixed random orthogonal B matrix, distilled at the early-exit layer with L2-norm-based gradient reassignment) and later blocks with task-specific LoRA scaled by learnable block-wise weights regularized to be orthogonal across tasks.
Key results
- On CIFAR-100 (T=20), CL-LoRA reaches 91.02% average accuracy vs. 91.63% for RanPAC (which uses 3.1% params vs. 0.3% for CL-LoRA).
- CL-LoRA achieves best final accuracy on ImageNet-R (T=10: 74.51%), ImageNet-A and VTAB (94.29%), among compared methods.
- Ablation on CIFAR-100 (T=10) shows KD with early exit, gradient reassignment and block weights together raise average accuracy from 88.20% to 91.85%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CL-LoRA instead of the strongest baseline for the reported metric | ViT-B/16 rehearsal-free CIL · CIFAR-100, T=20 | final accuracy: 87.62±0.16 -> 85.32±0.08 | Hurt |
| use CL-LoRA instead of the strongest baseline for the reported metric | ViT-B/16 rehearsal-free CIL · CIFAR-100, T=20 | average accuracy: 91.63±0.28 -> 91.02±0.12 | Hurt |
| use CL-LoRA instead of the strongest baseline for the reported metric | ViT-B/16 rehearsal-free CIL · ImageNet-R, T=40 | final accuracy: 71.43±0.18 -> 74.51±0.14 | Helped |
| use CL-LoRA instead of the strongest baseline for the reported metric | ViT-B/16 rehearsal-free CIL · ImageNet-R, T=40 | average accuracy: 78.53±0.73 -> 81.58±0.59 | Helped |
| use CL-LoRA instead of the strongest baseline for the reported metric | ViT-B/16 rehearsal-free CIL · ImageNet-A, T=10 | final accuracy: 59.25±0.88 -> 60.54±0.63 | Helped |
| use CL-LoRA instead of the strongest baseline for the reported metric | ViT-B/16 rehearsal-free CIL · ImageNet-A, T=10 | average accuracy: 68.92±2.06 -> 70.15±2.23 | Helped |
| use CL-LoRA instead of the strongest baseline for the reported metric | ViT-B/16 rehearsal-free CIL · VTAB, T=5 | final accuracy: 92.85±0.88 -> 94.29±0.34 | Helped |
| use CL-LoRA instead of the strongest baseline for the reported metric | ViT-B/16 rehearsal-free CIL · VTAB, T=5 | average accuracy: 93.01±0.33 -> 94.57±0.60 | Helped |
| add early-exit knowledge distillation (KD) (ablation) | CL-LoRA · CIFAR-100 and ImageNet-R | average accuracy: 88.20 -> 90.83; 88.13 -> 90.06; 82.24 -> 83.42; 79.61 -> 80.77 | Helped |
| add gradient reassignment (GR) to KD (ablation) | CL-LoRA · CIFAR-100 and ImageNet-R | average accuracy: 90.83 -> 91.69; 90.06 -> 90.72; 83.42 -> 84.08; 80.77 -> 81.25 | Helped |
| add block-wise weights (BW) (ablation) | CL-LoRA · CIFAR-100 and ImageNet-R | average accuracy: 88.20 -> 89.01; 88.13 -> 88.72; 82.24 -> 82.93; 79.61 -> 80.48 | Helped |
| combine KD, GR, and BW (ablation) | CL-LoRA · CIFAR-100 and ImageNet-R | average accuracy: 91.69 -> 91.85; 90.72 -> 91.02; 84.08 -> 84.77; 81.25 -> 81.58 | Helped |
Surprising findings
- A fixed random orthogonal down-projection outperformed a trainable down-projection at every split position, even when all adapters are task-specific (l=0).
- Increasing LoRA rank did not consistently improve accuracy; rank 1 already gave promising results on CIFAR-100.
Limitations
- Performance degrades when the shared portion l is too large, so the split position needs tuning.
- Does not provide an algorithm for selecting optimal LoRA rank or MHSA projection configuration per task.
Experimental setup
Datasets: CIFAR-100 ImageNet-R ImageNet-A VTAB
Benchmarks: CIFAR-100 ImageNet-R ImageNet-A VTAB
Metrics: average accuracy final accuracy trainable parameter percentage
Baselines: L2P DualPrompt CODA-Prompt LAE APER RanPAC EASE O-LoRA InfLoRA
Builds on: LoRA APER InfLoRA O-LoRA DualPrompt
Closest papers by meaning
- SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning (ICLR 2025 · similarity 0.93)
- C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models (NeurIPS 2026 · similarity 0.90)
- Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models (NeurIPS 2025 · similarity 0.90)
- Adapter Merging with Centroid Prototype Mapping for Scalable Class-Incremental Learning (CVPR 2025 · similarity 0.88)
- When the Merge Coefficient Stops Mattering: Proximity Regularized Merging for Continual LoRA Adaptation (NeurIPS 2026 · similarity 0.88)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.