Research map / Continual learning / Continual fine-tuning of LLMs
LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRanPAC, a continual learner that solves a minimum-norm least-squares problem on random ReLU features of frozen ViT features while continually truncating small SVD factors, giving stability and train/test error bounds.
Problem
Continual class-incremental learning with pre-trained models lacks theoretical guarantees, while theoretically grounded methods underperform; random-feature least-squares on lifted features is numerically unstable due to ill-conditioning, and RanPAC's ridge regularization is sensitive to its hyperpa
Approach
Continually truncate the smallest singular values of the lifted random ReLU features (via incremental truncated SVD) before solving an over-parameterized minimum-norm least-squares problem, replacing RanPAC's sensitive ridge regularization; a recurrence over eigenvalue gaps yields provable training and test MSE bounds.
Key results
- LoRanPAC averages 78.55% final accuracy across 8 datasets (B-0/B-10/B-16, Inc-5/10/20) versus 76.76% for RanPAC under the same setting.
- Under Inc-1 (one class at a time), LoRanPAC reaches 62.76% final accuracy on ImageNet-A where RanPAC gets 36.6% with high variance, and 74.32% on StanfordCars versus 35.56% for RanPAC.
- With first-session adaptation (EASE hyperparameters), LoRanPAC* averages 81.16% final accuracy, above EASE* at 76.07%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace RanPAC ridge regression with continual truncated-SVD least squares (LoRanPAC) | ImageNet-1K ViT continual classifier · CIFAR100, Inc-1 | final accuracy: 86.99$_{\pm0.06}$ -> 88.19$_{\pm0.05}$ | Helped |
| Replace RanPAC ridge regression with continual truncated-SVD least squares (LoRanPAC) | ImageNet-1K ViT continual classifier · ImageNet-R, Inc-1 | final accuracy: 70.12$_{\pm0.39}$ -> 73.66$_{\pm0.07}$ | Helped |
| Replace RanPAC ridge regression with continual truncated-SVD least squares (LoRanPAC) | ImageNet-1K ViT continual classifier · ImageNet-A, Inc-1 | final accuracy: 36.6$_{\pm25.35}$ -> 62.76$_{\pm0.16}$ | Helped |
| Replace RanPAC ridge regression with continual truncated-SVD least squares (LoRanPAC) | ImageNet-1K ViT continual classifier · CUB-200, Inc-1 | final accuracy: 55.15$_{\pm37.14}$ -> 89.19$_{\pm0.06}$ | Helped |
| Replace RanPAC ridge regression with continual truncated-SVD least squares (LoRanPAC) | ImageNet-1K ViT continual classifier · ObjectNet, Inc-1 | final accuracy: 57.14$_{\pm0.24}$ -> 60.82$_{\pm0.15}$ | Helped |
| Replace RanPAC ridge regression with continual truncated-SVD least squares (LoRanPAC) | ImageNet-1K ViT continual classifier · OmniBenchmark, Inc-1 | final accuracy: 77.9$_{\pm0.04}$ -> 79.3$_{\pm0.06}$ | Helped |
| Replace RanPAC ridge regression with continual truncated-SVD least squares (LoRanPAC) | ImageNet-1K ViT continual classifier · VTAB, Inc-1 | final accuracy: 91.47$_{\pm0.3}$ -> 92.51$_{\pm0.05}$ | Helped |
| Replace RanPAC ridge regression with continual truncated-SVD least squares (LoRanPAC) | ImageNet-1K ViT continual classifier · StanfordCars, Inc-1 | final accuracy: 35.56$_{\pm24.75}$ -> 74.32$_{\pm0.11}$ | Helped |
| Use random ReLU features instead of pre-trained features alone (ablation) | Joint linear classifier · CIFAR100 | final accuracy: 87.56 -> 87.76 | Helped |
| Use random ReLU features instead of pre-trained features alone (ablation) | Joint linear classifier · ImageNet-R | final accuracy: 72.42 -> 73.00 | Helped |
| Use random ReLU features instead of pre-trained features alone (ablation) | Joint linear classifier · ImageNet-A | final accuracy: 58.85 -> 59.25 | Helped |
| Use random ReLU features instead of pre-trained features alone (ablation) | Joint linear classifier · CUB-200 | final accuracy: 88.76 -> 88.72 | Hurt |
Surprising findings
- RanPAC's large gaps on ImageNet-A (Inc-5) and StanfordCars (Inc-5) stem from cross-validation failing to choose a suitable regularization lambda with small validation sets.
- Minimum-norm least squares on highly ill-conditioned random ReLU features fails numerically without truncation, with near-zero accuracy at truncation percentage zero.
Limitations
- Requires choosing a truncation percentage zeta, though authors say it is less sensitive than RanPAC's lambda.
- Theoretical bounds rely on a noisy linear model assumption; the authors note the linear model is assumed adequate empirically rather than proven for the full architecture.
- Adapter-based baselines such as EASE are infeasible for Inc-1 settings.
Experimental setup
Datasets: CIFAR100 ImageNet-R ImageNet-A CUB-200 ObjectNet OmniBenchmark VTAB StanfordCars
Benchmarks: CIFAR100 ImageNet-R ImageNet-A CUB-200 ObjectNet OmniBenchmark VTAB StanfordCars
Metrics: final accuracy total accuracy
Baselines: RanPAC L2P DualPrompt CodaPrompt SimpleCIL ADAM EASE LC (joint linear classifier)
Builds on: RanPAC Ideal Continual Learner PCA-OGD EASE PILOT
Closest papers by meaning
- LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learning (CVPR 2025 · similarity 0.90)
- SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning (ICLR 2025 · similarity 0.90)
- C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models (NeurIPS 2026 · similarity 0.89)
- Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.89)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.89)
- BiLoRA: Almost-Orthogonal Parameter Spaces for Continual Learning (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.