Research map / Continual learning / Continual fine-tuning of LLMs
Learn more, but bother less: parameter efficient continual learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LB-CL, a parameter-efficient continual learning method for T5 models that initializes new SVD-based low-rank adapters from sensitivity-weighted previous-task triplets and trains them with gradient projection onto previous subspaces.
Problem
Continual learning in LLMs suffers catastrophic forgetting, and existing parameter-efficient continual learning methods mostly ignore forward knowledge transfer from earlier tasks to new ones.
Approach
Extract sensitivity-weighted SVD triplets (singular value and vectors) from previous tasks' adapters to initialize the new task's adapter (knowledge injection), then project new-task gradients orthogonal to previous low-rank subspaces to limit forgetting.
Key results
- On the standard CL benchmark with T5-large, LB-CL averages 76.7 accuracy across three orders, versus 75.4 for O-LoRA.
- On the 15-task long-sequence benchmark, LB-CL averages 69.2 versus 68.8 for O-LoRA; ProgPrompt (78.4) and PerTaskFT/MTL remain higher.
- Ablations show that both initialization and gradient projection contribute; LB-CL with T5-large reaches consistent scores across orders (76.5 to 76.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LB-CL instead of O-LoRA | T5-large continual learning · Standard CL Benchmark, avg | accuracy: 75.4 -> 76.7 | Helped |
| use LB-CL instead of O-LoRA | T5-large continual learning · Large Number of Tasks, avg | accuracy: 68.8 -> 69.2 | Helped |
| add gradient projection to knowledge-transfer initialization (ablation) | LB-CL · Standard CL Benchmark, avg | accuracy: 73.6 -> 76.7 | Helped |
| add knowledge-transfer initialization to gradient projection (ablation) | LB-CL · Standard CL Benchmark, avg | accuracy: 74.3 -> 76.7 | Helped |
| add both knowledge-transfer initialization and gradient projection (ablation) | LB-CL · Large Number of Tasks, avg | accuracy: 65.2 -> 69.2 | Helped |
| omit knowledge-transfer initialization and gradient projection (ablation) | LB-CL · Standard CL Benchmark, avg | accuracy: 76.7 -> 74.5 | Hurt |
| remove singular values from the initialization triplets, retaining U and V (ablation) | LB-CL initialization · Standard CL Benchmark | accuracy | Helped |
| increase seed samples used for sensitivity scoring (ablation) | LB-CL · Standard CL Benchmark | accuracy | Helped |
| vary low-rank matrix rank from 2 to 16 (ablation) | LB-CL · Standard CL Benchmark, avg | accuracy: 76.3, 76.6, 76.7, 76.3 | No clear effect |
| scale the base model from T5-base to T5-large | LB-CL · Standard CL Benchmark, avg | accuracy: 73.5 -> 76.7 | Helped |
Surprising findings
- NLNB-CL, with neither initialization nor gradient projection, performs slightly better on average than L-CL and B-CL.
- Initialization using only U and V from prior triplets (without Sigma) outperforms the full triplet configuration on average.
Limitations
- Limitations section is absent; the checklist says limitations are not discussed.
- Evaluation limited to T5-base and T5-large on text classification tasks; no decoder-only LLMs tested.
- Results reported without error bars or significance tests; the gains over O-LoRA are small (about 1.3 points on the standard benchmark, 0.4 on long-sequence).
Experimental setup
Datasets: AG News Amazon Reviews Yelp Reviews DBpedia Yahoo Answers MNLI QQP RTE SST-2 WiC CB COPA
Benchmarks: Standard CL benchmark (AG News, Amazon, Yelp, DBpedia, Yahoo; 3 orders) 15-dataset long-sequence CL benchmark (3 orders)
Metrics: Average Accuracy forgetting ROUGE-L
Baselines: SeqFT SeqLoRA IncLoRA SeqSVD Replay EWC LwF L2P LFPT5 O-LoRA ProgPrompt PerTaskFT
Builds on: O-LoRA AdaLoRA LoRA Orthogonal gradient descent LFPT5
Closest papers by meaning
- Scalable Language Model with Generalized Continual Learning (ICLR 2024 · similarity 0.92)
- Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models (NeurIPS 2026 · similarity 0.91)
- Spurious Forgetting in Continual Learning of Language Models (ICLR 2025 · similarity 0.91)
- Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.91)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.90)
- Train-Attention: Meta-Learning Where to Focus in Continual Knowledge Learning (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.