Research map / Continual learning / Continual fine-tuning of LLMs
InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InfLoRA, a PEFT-based continual learning method that constrains each new task's LoRA down-projection to a subspace orthogonal to old-task gradient spaces (via DualGPM) so new-task updates don't interfere with old tasks.
Problem
Class-incremental continual learning with frozen pre-trained ViT and PEFT modules, where learning new tasks interferes with old tasks, hurting the stability-plasticity trade-off.
Approach
Reparameterize pre-trained weights with a LoRA branch whose down-projection B_t is pre-designed (via SVD of new-task inputs projected onto the orthogonal complement of old-task gradient space), so only A_t is trained; this makes fine-tuning equivalent to updating W within a subspace that does not interfere with old tasks.
Key results
- On ImageNet-R with 10 tasks, InfLoRA reaches 82.01% final accuracy? (5 tasks: 77.52 ACC5; 10 tasks: 80.82 ACC10 average accuracy) outperforming the best baseline CODA-P at 75.82 ACC10 average.
- On CIFAR100 (10 tasks), InfLoRA achieves 91.70% ACC10 average, versus 91.08% for CODA-P.
- On DomainNet (5 tasks), InfLoRA achieves 79.57% ACC5 average, versus 78.72% for CODA-P.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InfLoRA instead of C-LoRA | continual learning with ViT-B/16 · ImageNet-R (10 tasks) | ACC10: 71.89 ± 0.45 -> 75.65 ± 0.14 | Helped |
| Use InfLoRA instead of CODA-P | continual learning with ViT-B/16 · ImageNet-R (10 tasks) | averaged ACC10: 75.82 ± 0.29 -> 80.82 ± 0.24 | Helped |
| Use InfLoRA instead of CODA-P | continual learning with ViT-B/16 · CIFAR100 (10 tasks) | ACC10: 86.62 ± 0.11 -> 86.51 ± 0.73 | Hurt |
| Use InfLoRA instead of CODA-P | continual learning with ViT-B/16 · CIFAR100 (10 tasks) | averaged ACC10: 91.08 ± 0.28 -> 91.70 ± 0.32 | Helped |
| Use InfLoRA instead of CODA-P | continual learning with ViT-B/16 · DomainNet (5 tasks) | ACC5: 73.23 ± 0.13 -> 74.53 ± 0.23 | Helped |
| Use InfLoRA instead of CODA-P | continual learning with ViT-B/16 · DomainNet (5 tasks) | averaged ACC5: 78.72 ± 0.07 -> 79.57 ± 0.57 | Helped |
| Design Bt by projecting new-task inputs into M⊥t and selecting top-r principal components, rather than random Gaussian initialization (ablation) | InfLoRA · ImageNet-R (10 tasks) | ACC10: 67.38 ± 0.41 -> 75.65 ± 0.14 | Helped |
| Add the projection onto M⊥t when designing Bt from Nt (ablation) | InfLoRA · ImageNet-R (10 tasks) | ACC10: 57.91 ± 0.30 -> 75.65 ± 0.14 | Helped |
| Add the new-task subspace Nt when designing Bt from M⊥t (ablation) | InfLoRA · ImageNet-R (10 tasks) | ACC10: 74.61 ± 0.62 -> 75.65 ± 0.14 | Helped |
Limitations
- Constraining updates to a shrinking subspace may reduce capacity for later tasks as the number of tasks grows.
- Exemplar-free setting only; evaluated mainly on ViT-B/16 image classification benchmarks.
Experimental setup
Datasets: ImageNet-R CIFAR100 DomainNet
Benchmarks: ImageNet-R CIFAR100 DomainNet
Metrics: final accuracy ACC_T average accuracy accuracy over tasks
Baselines: L2P DualPrompt CODA-P C-LoRA LAE SLCA joint sequential
Builds on: LoRA DualGPM Gradient Projection Memory L2P DualPrompt CODA-P
Closest papers by meaning
- BiLoRA: Almost-Orthogonal Parameter Spaces for Continual Learning (CVPR 2025 · similarity 0.90)
- SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models (NeurIPS 2024 · similarity 0.89)
- Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models (NeurIPS 2025 · similarity 0.89)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.88)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.88)
- Large Continual Instruction Assistant (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.