Research map / Continual learning / Continual fine-tuning of LLMs
Continuous Subspace Optimization for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoSO, which fine-tunes pre-trained ViTs in a sequence of gradient-SVD-derived subspaces, keeps each task's subspace orthogonal to a historical task subspace built via Frequent Directions sketching, and outperforms PEFT continual learning baselines on ImageNet-R, CIFAR100, and DomainNet.
Problem
Continual learning with pre-trained ViTs suffers from catastrophic forgetting, and existing LoRA-based methods restrict updates to a fixed low-rank subspace, limiting learning capacity.
Approach
Optimize in a sequence of dynamically chosen gradient subspaces (GaLore-style SVD projection with Adam) that are kept orthogonal to the accumulated historical task subspace; a task-specific subspace is estimated online with Frequent Directions sketching and then merged into the historical basis after each task.
Key results
- On ImageNet-R 20 tasks, CoSO reaches 78.19% final accuracy vs 75.42% for the best baseline (VPT-NSP^2).
- On DomainNet (5 tasks), CoSO achieves 74.27% final accuracy, 1.75 points above the best baseline.
- Removing orthogonal projection drops ImageNet-R 20-task final accuracy by 8.52 points in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CoSO instead of the strongest baseline, VPT-NSP^2 | ViT-B/16 continual learning · ImageNet-R (5 Tasks) | final accuracy: 79.72 -> 82.10 | Helped |
| use CoSO instead of the strongest baseline, VPT-NSP^2 | ViT-B/16 continual learning · ImageNet-R (10 Tasks) | final accuracy: 77.87 -> 81.10 | Helped |
| use CoSO instead of the strongest baseline, VPT-NSP^2 | ViT-B/16 continual learning · ImageNet-R (20 Tasks) | final accuracy: 75.42 -> 78.19 | Helped |
| use CoSO instead of the strongest baseline, VPT-NSP^2 | ViT-B/16 continual learning · CIFAR100 (10 Tasks) | final accuracy: 88.09 -> 88.77 | Helped |
| use CoSO instead of the strongest baseline, SD-LoRA | ViT-B/16 continual learning · DomainNet (5 Tasks) | final accuracy: 73.20 -> 74.27 | Helped |
| add orthogonal projection to exclude historical task directions (ablation) | CoSO · ImageNet-R (20 Tasks) | final accuracy: 69.75 -> 78.27 | Helped |
| add orthogonal projection to exclude historical task directions (ablation) | CoSO · ImageNet-R (5 Tasks) | final accuracy: 79.35 -> 82.37 | Helped |
| add orthogonal projection to exclude historical task directions (ablation) | CoSO · ImageNet-R (10 Tasks) | final accuracy: 75.90 -> 80.72 | Helped |
| use FD to consolidate intermediate task gradients instead of only the final task subspace (ablation) | CoSO · ImageNet-R (5 Tasks) | final accuracy: 80.72 -> 82.37 | Helped |
| use FD to consolidate intermediate task gradients instead of only the final task subspace (ablation) | CoSO · ImageNet-R (10 Tasks) | final accuracy: 78.83 -> 80.72 | Helped |
| use FD to consolidate intermediate task gradients instead of only the final task subspace (ablation) | CoSO · ImageNet-R (20 Tasks) | final accuracy: 76.68 -> 78.27 | Helped |
Surprising findings
- The w/o Orth ablation on ImageNet-R (20 tasks) falls below several baselines, indicating orthogonality is critical.
Limitations
- Only vision classification benchmarks evaluated; extension to multimodal settings left open.
- Memory footprint slightly higher than InfLoRA due to larger sketch rank.
- Requires several hyperparameters (r1, r2, K, threshold) tuned per dataset.
Experimental setup
Datasets: ImageNet-R CIFAR100 DomainNet
Benchmarks: ImageNet-R (5, 10, 20 tasks) CIFAR100 (10 tasks) DomainNet (5 tasks)
Metrics: final accuracy average accuracy
Baselines: L2P DualPrompt CODA-Prompt InfLoRA VPT-NSP^2 SD-LoRA
Builds on: GaLore InfLoRA Adam Frequent Directions VPT-NSP^2
Closest papers by meaning
- BiLoRA: Almost-Orthogonal Parameter Spaces for Continual Learning (CVPR 2025 · similarity 0.89)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.88)
- An Effective Dynamic Gradient Calibration Method for Continual Learning (ICML 2024 · similarity 0.87)
- Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.87)
- LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning (ICLR 2025 · similarity 0.87)
- Continual Learning with Global Alignment (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.