Research map / Continual learning / Continual fine-tuning of LLMs
Low-rank and model-merging adaptation
Controls forgetting in continual fine-tuning through low-rank updates, prompts and weight merging.
90 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2019–2026) · an approach in Continual fine-tuning of LLMs, keyword cluster “pre trained · lora · fine tuning”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
17 papers in 2023–24 and 69 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 32.7% to 35.4%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2019 · Catastrophic Forgetting Meets Negative Transfer: Batch Spectral Shrinkage for Safe Transfer Learning (NeurIPS 2019 · 101 citations)
- 2021 · Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (NeurIPS 2021 · 43 citations)
- 2022 · Memory Efficient Continual Learning with Transformers (NeurIPS 2022 · 10 citations)
- 2023 · RanPAC: Random Projections and Pre-trained Models for Continual Learning (NeurIPS 2023 · 23 citations)
- 2024 · Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters (CVPR 2024 · 124 citations)
- 2025 · Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning (CVPR 2025 · 18 citations)
- 2026 · LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuning (AAAI 2026 · 2 citations)
Earliest papers here
- Catastrophic Forgetting Meets Negative Transfer: Batch Spectral Shrinkage for Safe Transfer Learning (NeurIPS 2019 · 101 citations)
- Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting (ICML 2019 · 73 citations)
- Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (NeurIPS 2021 · 43 citations)
What has been tried
37 papers here report 437 measured changes (a component added, removed or swapped, compared with a baseline): helped 71%, no clear effect 4%, hurt 22%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add task-specific routers (11 routers rather than one) | MTIL (Transfer) · accuracy | 44.1 -> 68.6 | Helped | Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters (CVPR 2024 · 124 citations) |
| Design Bt by projecting new-task inputs into M⊥t and selecting top-r principal components, rather than random Gaussian initialization | ImageNet-R (10 tasks) · ACC10 | 67.38 ± 0.41 -> 75.65 ± 0.14 | Helped | InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning (CVPR 2024 · 69 citations) |
| Add shared embeddings across tasks | ImageNet-R, 10 tasks · average accuracy (AT) | 62.1 ± 0.52 -> 67.3 ± 0.58 | Helped | Convolutional Prompting meets Language Models for Continual Learning (CVPR 2024 · 20 citations) |
| Replace adapter tuning with SSF | CIFAR100 · average session accuracy | 91.52 -> 83.03 | Hurt | Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer (CVPR 2024 · 20 citations) |
| add self-expansion rather than keeping only first-session adaptation | ImageNet-A · average accuracy at last step | 49.90 -> 53.32 | Helped | Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning (CVPR 2025 · 18 citations) |
| use LoRA Subtraction to construct DRS instead of training with LoRA alone | ImageNet-R, 50 tasks · average accuracy | 58.56 -> 76.03 | Helped | LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learning (CVPR 2025 · 12 citations) |
| allocate distinct rather than shared frequency coordinates across tasks | CIFAR-100, ImageNet-R, DomainNet · Last and average accuracy | CIFAR-100: 74.42 / 77.52 -> 75.65 / 81.52; ImageNet-R: 84.46 / 89.15 -> 87.46 / 92.50; DomainNet: 74.15 / 78.97 -> 76.56 | Helped | BiLoRA: Almost-Orthogonal Parameter Spaces for Continual Learning (CVPR 2025 · 10 citations) |
| use tanh activation in NoRGa | Split CIFAR-100 (Sup-21K and iBOT-21K) · FA | 92.61 -> 94.36; 93.02 -> 94.76 | Helped | Mixture of Experts Meets Prompt-Based Continual Learning (NeurIPS 2024 · 8 citations) |
| Add the slow learner to the baseline | ImageNet-A · final accuracy | 62.21 -> 65.44 | Helped | SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models (NeurIPS 2024 · 4 citations) |
| add gradient projection to knowledge-transfer initialization | Standard CL Benchmark, avg · accuracy | 73.6 -> 76.7 | Helped | Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · 3 citations) |
| Replace random parameter selection with adaptive parameter selection | Flickr30k across drop rates · target task performance | Helped | AdaDARE-gamma: Balancing Stability and Plasticity in Multi-modal LLMs through Efficient Adaptation (CVPR 2025 · 2 citations) | |
| Add test-time adaptation to fixed-weight low-rank experts | ViT-B/16, 8 tasks · ACC (%) | 78.7 -> 86.4 | Helped | MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging (NeurIPS 2025 · 2 citations) |
What reviewers will expect
Datasets and baselines named in the 30 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-100 · 11 of 30 papers
- ImageNet-R · 10 of 30 papers
- DomainNet · 6 of 30 papers
- ImageNet-A · 5 of 30 papers
- CUB-200 · 4 of 30 papers
- VTAB · 2 of 30 papers
- Flowers102 · 2 of 30 papers
- Stanford Cars · 2 of 30 papers
Baselines
- L2P · 10 of 30 papers
- DualPrompt · 10 of 30 papers
- InfLoRA · 10 of 30 papers
- CODA-Prompt · 8 of 30 papers
- LoRA · 5 of 30 papers
- LAE · 4 of 30 papers
- O-LoRA · 4 of 30 papers
- SD-LoRA · 4 of 30 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 86 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 4 papers
- Nanjing University · 3 papers
- Harbin Institute of Technology · 3 papers
- Xidian University · 2 papers
- University of Electronic Science and Technology of China · 2 papers
- Amazon · 2 papers
- Nanyang Technological University · 2 papers
- Sun Yat-sen University · 2 papers
- University of New South Wales · 2 papers
- National University of Singapore · 2 papers
Use it
Download BibTeX of 27 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Synthetic data and merging for CLIP (32 papers)
Elsewhere in the topic
- Continual pretraining of language models (66 papers)
- Mixture-of-experts for multimodal continual tuning (52 papers)
- Machine unlearning by selective forgetting (24 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).