Research map / Continual learning / Continual fine-tuning of LLMs
BiLoRA: Almost-Orthogonal Parameter Spaces for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BiLoRA, a continual learning method that replaces learned LoRA subspaces with sparse, fixed-basis (Fourier/DFT) bilinear updates whose random frequency supports are almost orthogonal across tasks.
Problem
Class-incremental continual learning with PEFT (LoRA) suffers task interference; InfLoRA's learned orthogonal subspaces limit the number of tasks to about d/r and require complex gradient-based allocation.
Approach
Fix orthogonal bases U and V (here the 2D DFT) and learn a sparse set of k frequency coefficients per task, with task supports sampled randomly (with or without replacement) so that bilinear expansion to d^2 dimensions makes task updates almost orthogonal with high probability, avoiding explicit orthogonalization.
Key results
- On ImageNet-R (10 tasks), BiLoRA reaches 77.95% last and 81.52% average accuracy, versus 75.65% and 80.82% for InfLoRA.
- On CIFAR-100 (10 tasks), BiLoRA reaches 87.46% last accuracy against 91.92% joint training.
- On ImageNet-R with 20 tasks, BiLoRA achieves 72.41% last accuracy versus 71.01% for InfLoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use BiLoRA instead of InfLoRA | ViT-B/16 continual learning · ImageNet-R (10 Task) | Last accuracy: 75.65 ± 0.19 -> 77.95 ± 0.14 | Helped |
| use BiLoRA instead of InfLoRA | ViT-B/16 continual learning · CIFAR-100 (10 Task) | Last accuracy: 86.51 ± 0.73 -> 87.46 ± 0.76 | Helped |
| use BiLoRA instead of InfLoRA | ViT-B/16 continual learning · DomainNet (10 Task) | Last accuracy: 75.45 ± 0.22 -> 76.56 ± 0.41 | Helped |
| allocate distinct rather than shared frequency coordinates across tasks (ablation) | BiLoRA · CIFAR-100, ImageNet-R, DomainNet | Last and average accuracy: CIFAR-100: 74.42 / 77.52 -> 75.65 / 81.52; ImageNet-R: 84.46 / 89.15 -> 87.46 / 92.50; DomainNet: 74.15 / 78.97 -> 76.56 | Helped |
| sample frequency coordinates without replacement instead of randomly with replacement (ablation) | BiLoRA · CIFAR-100, ImageNet-R, DomainNet | Last and average accuracy: CIFAR-100: 75.65 / 81.52 -> 75.72 / 81.65; ImageNet-R: 87.46 / 92.50 -> 87.48 / 92.60; DomainNet: 76.56 / 81.65 -> 76.74 | Helped |
| set k=1000 rather than k=3000 (ablation) | BiLoRA · CIFAR-100, ImageNet-R, DomainNet | Average accuracy: CIFAR-100: 85.2 -> 87.46; ImageNet-R: 79.7 -> 81.52; DomainNet: 75.1 -> 76.56 | Hurt |
| increase frequency components from k=3000 to k=6000 (ablation) | BiLoRA · CIFAR-100, ImageNet-R, DomainNet | Average accuracy: CIFAR-100: 87.46 -> 87.30; ImageNet-R: 81.52 -> 81.48; DomainNet: 76.56 -> 76.23 | Hurt |
| increase frequency components from k=3000 to k=12000 (ablation) | BiLoRA · CIFAR-100, ImageNet-R, DomainNet | Average accuracy: CIFAR-100: 87.46 -> 87.30; ImageNet-R: 81.52 -> 81.48; DomainNet: 76.56 -> 76.23 | Hurt |
| increase frequency components from k=3000 to k=24000 (ablation) | BiLoRA · CIFAR-100, ImageNet-R, DomainNet | Average accuracy: CIFAR-100: 87.46 -> 87.30; ImageNet-R: 81.52 -> 81.48; DomainNet: 76.56 -> 76.23 | Hurt |
| increase frequency components from k=3000 to k=48000 (ablation) | BiLoRA · CIFAR-100, ImageNet-R, DomainNet | Average accuracy: CIFAR-100: 87.46 -> 87.30; ImageNet-R: 81.52 -> 81.48; DomainNet: 76.56 -> 76.23 | Hurt |
Surprising findings
- Random sampling without replacement gives essentially identical accuracy to sampling with replacement, and the paper's ablation shows performance plateaus beyond a small sparsity level.
Limitations
- Theoretical bounds are stated as conservative and the interference analysis relies on simulation rather than tight proofs.
- Evaluation limited to class-incremental image classification on three datasets with a single ViT-B/16 backbone.
- Analysis only covers LoRA-based architectures; generalization to other models is untested.
Experimental setup
Datasets: ImageNet-R CIFAR-100 DomainNet
Benchmarks: ImageNet-R CIFAR-100 DomainNet
Metrics: last accuracy average accuracy
Baselines: Sequential L2P DualPrompt CODA-P C-LoRA LAE InfLoRA
Builds on: LoRA InfLoRA L2P DualPrompt CODA-P
Closest papers by meaning
- InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning (CVPR 2024 · similarity 0.90)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.90)
- Continuous Subspace Optimization for Continual Learning (NeurIPS 2025 · similarity 0.89)
- LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning (ICLR 2025 · similarity 0.89)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.88)
- A Statistical Theory of Regularization-Based Continual Learning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.