Research map / Continual learning / Continual fine-tuning of LLMs
Continual pretraining of language models
Analyzes and mitigates forgetting during continual pretraining and knowledge updating of large models.
66 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Continual fine-tuning of LLMs, keyword cluster “unlearning · multimodal · instruction”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
16 papers in 2023–24 and 41 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 30.8% to 21.0%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Memory-based Parameter Adaptation (ICLR 2018 · 26 citations)
- 2020 · LAMOL: LAnguage MOdeling for Lifelong Language Learning (ICLR 2020 · 111 citations)
- 2022 · Towards Continual Knowledge Learning of Language Models (ICLR 2022 · 39 citations)
- 2023 · Continual Pre-training of Language Models (ICLR 2023 · 16 citations)
- 2024 · Interactive Continual Learning: Fast and Slow Thinking (CVPR 2024 · 20 citations)
- 2025 · Language Guided Concept Bottleneck Models for Interpretable Continual Learning (CVPR 2025 · 10 citations)
- 2026 · Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Models (ICML 2026 · 0 citations)
Earliest papers here
- Memory-based Parameter Adaptation (ICLR 2018 · 26 citations)
- LAMOL: LAnguage MOdeling for Lifelong Language Learning (ICLR 2020 · 111 citations)
- Towards Continual Knowledge Learning of Language Models (ICLR 2022 · 39 citations)
What has been tried
36 papers here report 384 measured changes (a component added, removed or swapped, compared with a baseline): helped 69%, no clear effect 5%, hurt 22%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add System2 collaborative inference with Pure-MM | CIFAR10 Class-IL · accuracy | 95.54 -> 96.83 | Helped | Interactive Continual Learning: Fast and Slow Thinking (CVPR 2024 · 20 citations) |
| use 100 concepts per task instead of 10 | CUB-200 B-10 Inc-10 · average incremental accuracy | 79.32 -> 86.07 | Helped | Language Guided Concept Bottleneck Models for Interpretable Continual Learning (CVPR 2025 · 10 citations) |
| update attention layers instead of first MLP layers | 6 CIL sequences (average) · Acc. In. | 21.34 -> 20.21 | Hurt | Overcoming Generic Knowledge Loss with Selective Parameter Update (CVPR 2024 · 10 citations) |
| Add the review stage | Original languages · average score | 47.14 -> 52.12 | Helped | MoE-LPR: Multilingual Extension of Large Language Models Through Mixture-of-Experts with Language Priors Routing (AAAI 2025 · 9 citations) |
| Add Memory-Aware Objective | StreamingQA · EM | 18.54 -> 18.36 | No clear effect | CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Models (AAAI 2025 · 4 citations) |
| replace JARe weight increments with prompts | BERT benchmark (Orders 4–6, Avg) · accuracy | 79.0 -> 53.3 | Hurt | Scalable Language Model with Generalized Continual Learning (ICLR 2024 · 2 citations) |
| add patch decoration (weight compensation) | Flickr30k target · H-score | 91.96 -> 93.67 | Helped | Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models (ICML 2024 · 2 citations) |
| Add intra-task memory X_cfg | Spider-stream-semi · ACC_a (EM) | 45.6 -> 47.4 | Helped | Filling Memory Gaps: Enhancing Continual Semantic Parsing via SQL Syntax Variance-Guided LLMs Without Real Data Replay (AAAI 2025 · 1 citation) |
| Replace the continuous stream of novel data with repeated training on the same 100M tokens after the second epoch | Mixed seven-benchmark evaluation · accuracy difference | Hurt | How Much Can We Forget about Data Contamination? (ICML 2025 · 1 citation) | |
| Add concept-label replay to experience replay | CIFAR-100, CUB200, and ImageNet-100 · linear accuracy | 0.4511 -> 0.7650; 0.1382 -> 0.7914; 0.5903 -> 0.7722 | Helped | Walking the Web of Concept-Class Relationships in Incrementally Trained Interpretable Models (AAAI 2025 · 0 citations) |
| add LR-weighted coefficient ε=0.1 to S_2 | D_pt (FineWeb) and D_cpt (Knowledge Pile) · Huber Loss | D_pt: 0.0016 -> 0.0016; D_cpt: 0.0021 -> 0.0025 | Mixed | Learning Dynamics in Continual Pre-Training for Large Language Models (ICML 2025 · 0 citations) |
| replace stable EMA (β=0.99) with dynamical EMA | CoIN benchmark · Avg.ACC | +7.24@Avg.ACC | Helped | Large Continual Instruction Assistant (ICML 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 24 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CUB-200 · 3 of 24 papers
- Spider · 2 of 24 papers
- SlimPajama · 2 of 24 papers
- CIFAR-100 · 2 of 24 papers
- Tiny-ImageNet · 2 of 24 papers
- ARC-Easy · 2 of 24 papers
- WinoGrande · 2 of 24 papers
- PiQA · 2 of 24 papers
Baselines
- EWC · 5 of 24 papers
- SFT · 2 of 24 papers
- EMAR · 2 of 24 papers
- SFNET · 2 of 24 papers
- C3 · 2 of 24 papers
- ProgPrompt · 2 of 24 papers
- ICICLE · 2 of 24 papers
- L2P · 2 of 24 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 57 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 5 papers
- Harbin Institute of Technology · 4 papers
- Alibaba Group · 3 papers
- Peking University · 3 papers
- Zhejiang University · 3 papers
- Chinese Academy of Sciences · 2 papers
- Peng Cheng Laboratory · 2 papers
- Hong Kong University of Science and Technology · 2 papers
- Google · 2 papers
- Samsung · 2 papers
Use it
Download BibTeX of 30 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Mixture-of-experts for multimodal continual tuning (52 papers)
- Machine unlearning by selective forgetting (24 papers)
Elsewhere in the topic
- Low-rank and model-merging adaptation (90 papers)
- Synthetic data and merging for CLIP (32 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).