Research map / Continual learning / Continual fine-tuning of LLMs

PaLoRA: Paced Low-Rank Adaptation for Continual Learning

NeurIPS 2026 · Continual learning › Continual fine-tuning of LLMs · approach: Low-rank and model-merging adaptation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PaLoRA, a LoRA-based example-free continual learning method that compresses past updates via adaptive SVD, projects gradients onto the historical nullspace, and scales LoRA B-gradients by a rank-aware pacing factor sqrt(R/C), derived from a leakage-based budget model.

Problem

Catastrophic forgetting in LoRA-based continual learning: existing methods constrain update directions and use small learning rates as an unexplained magnitude heuristic, which does not adapt as the effective rank of accumulated knowledge grows.

Approach

Derives a pacing law s*=sqrt(R/c) for the gradient-step denominator under an anisotropic leakage model and budget constraint, where R is the effective rank of past updates, and instantiates it with adaptive SVD truncation, nullspace projection, and rank-aware gradient scaling plus a triplet-style regularizer.

Key results

• At 50 tasks on ImageNet-R, PaLoRA reaches 76.56% final accuracy, 4.09 points above LoRA-DRS (five seeds).
• At 50 tasks on ImageNet-A, PaLoRA reaches 60.90% final accuracy, 3.99 points above the strongest baseline.
• On CIFAR-100 at 50 tasks, PaLoRA reaches 88.96±0.14% final accuracy, outperforming LoRA-DRS by 1.94 points.

Surprising findings

• Adding the triplet loss to plain Sequence reduces 50-task CIFAR-100 final accuracy to 80.00%, while within PaLoRA it adds a small gain.
• The pacing-law optimality claim is stated to apply only to the budget-constrained linear model, not to the full regularized deep ViT system.

Limitations

• Pacing law derived for single-layer linear model under assumed anisotropic leakage and local quadratic sensitivity; deep ViT application is approximate.
• Evaluation limited to image classification; generative models not tested.
• Triplet regularizer is empirically motivated and not covered by the theory.

Experimental setup

Datasets: CIFAR-100 ImageNet-R ImageNet-A

Benchmarks: CIFAR-100 ImageNet-R ImageNet-A

Metrics: final accuracy average incremental accuracy backward transfer

Baselines: Sequence Pretrain L2P DualPrompt CODA-Prompt InfLoRA BiLoRA LoRA-DRS SD-LoRA

Builds on: LoRA SLCA InfLoRA O-LoRA GPM

Closest papers by meaning

• C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models (NeurIPS 2026 · similarity 0.92)
• Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.90)
• LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learning (CVPR 2025 · similarity 0.90)
• STAR: Stability-Inducing Weight Perturbation for Continual Learning (ICLR 2025 · similarity 0.89)
• SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning (ICLR 2025 · similarity 0.89)
• BECAME: Bayesian Continual Learning with Adaptive Model Merging (ICML 2025 · similarity 0.89)