Research map / Continual learning / Online and replay-based
Theoretical analysis of forgetting
Studies why and when networks forget, using linear models, teacher-student setups and complexity results.
141 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Online and replay-based, keyword cluster “weight · transfer · growing”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Few recent reports The tracked benchmarks have at most one report from 2025–26 that could beat the old record, too few to say whether they are still moving.
36 papers in 2023–24 and 55 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 22.1% to 27.9%.
Open questions the data points to
- Papers report GPM on Split miniImageNet (BWT (%)) anywhere from -0.7 to 0.3: comparisons against it depend on whose number you take.
No longer reported by papers here: Split CIFAR-100 (BWT (%), last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Continual Learning Through Synaptic Intelligence (ICML 2017 · 1,007 citations)
- 2018 · Overcoming Catastrophic Forgetting with Hard Attention to the Task (ICML 2018 · 387 citations)
- 2019 · Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning (CVPR 2019 · 296 citations)
- 2020 · Understanding the Role of Training Regimes in Continual Learning (NeurIPS 2020 · 94 citations)
- 2021 · On Learning the Geodesic Path for Incremental Learning (CVPR 2021 · 123 citations)
- 2022 · Probing Representation Forgetting in Supervised and Unsupervised Continual Learning (CVPR 2022 · 52 citations)
- 2023 · Achieving a Better Stability-Plasticity Trade-Off via Auxiliary Networks in Continual Learning (CVPR 2023 · 42 citations)
- 2024 · Divide and not forget: Ensemble of selectively trained experts in Continual Learning (ICLR 2024 · 11 citations)
- 2025 · Rethinking and Improving Student Learning and Forgetting Processes for Attention based Knowledge Tracing Models (AAAI 2025 · 17 citations)
- 2026 · Catastrophic Forgetting in Kolmogorov-Arnold Networks (AAAI 2026 · 1 citation)
Earliest papers here
- Continual Learning Through Synaptic Intelligence (ICML 2017 · 1,007 citations)
- Gradient Episodic Memory for Continual Learning (NeurIPS 2017 · 493 citations)
- Overcoming Catastrophic Forgetting by Incremental Moment Matching (NeurIPS 2017 · 277 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
Split CIFAR-100 · ACC (%) (higher is better)
6 papers, 2022–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 75.54 | Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer (NeurIPS 2022 · 8 citations) |
| 2023 | 81.4 | Adaptive Plasticity Improvement for Continual Learning (CVPR 2023 · 19 citations) |
Split CIFAR-100 · BWT (%) (higher is better)
5 papers, 2022–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 0.13 | Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer (NeurIPS 2022 · 8 citations) |
What has been tried
101 papers here report 1,125 measured changes (a component added, removed or swapped, compared with a baseline): helped 63%, no clear effect 8%, hurt 23%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add KERPLE-Power while retaining RKT's forgetting module | AL2005 · AUC | 0.7761±0.0010 -> 0.7837±0.0008 | Helped | Rethinking and Improving Student Learning and Forgetting Processes for Attention based Knowledge Tracing Models (AAAI 2025 · 17 citations) |
| Share more layers across experts | CIFAR-100 task-aware, 20-split and 50-split · average task-aware accuracy (%) | 86.8±0.3 -> 85.6±0.3; 91.2±0.4 -> 89.6±0.2 | Hurt | Divide and not forget: Ensemble of selectively trained experts in Continual Learning (ICLR 2024 · 11 citations) |
| Increase the regularization coefficient λ | 10-Split CIFAR-100 · forgetting | Hurt | BrainWash: A Poisoning Attack to Forget in Continual Learning (CVPR 2024 · 8 citations) | |
| Apply C-Flat in only 10% of iterations | CIFAR-100 B0_Inc20 · accuracy | 60.28 -> 60.48 | Helped | Make Continual Learning Stronger via C-Flat (NeurIPS 2024 · 7 citations) |
| Increase memory-buffer allocation from nc=10 to nc=100 | Aromaticity-CL · AP | 56.00±7.85 -> 68.24±5.68 | Helped | Exploring Rationale Learning for Continual Graph Learning (AAAI 2025 · 4 citations) |
| randomize task ordering while keeping the optimal replay fraction | synthetic continual learning · average loss | Hurt | Optimal Protocols for Continual Learning via Statistical Physics and Control Theory (ICLR 2025 · 4 citations) | |
| replace EAE with a fully connected classifier trained without an orthogonal projector | Split MNIST · accuracy | 96.62 -> 19.96 | Hurt | Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding (AAAI 2024 · 4 citations) |
| Use dynamic preference adaptation rather than fixed inference preference (0.5, 0.5) | Seq-CIFAR100 online · average incremental accuracy | Helped | Pareto Continual Learning: Preference-Conditioned Learning and Adaption for Dynamic Stability-Plasticity Trade-off (AAAI 2025 · 2 citations) | |
| increase context neurons per minicolumn (Nk) | Offline sequence capacity · maximum sequence length | Helped | Predictive Attractor Models (NeurIPS 2024 · 2 citations) | |
| Add replay without strong augmentation or SSL | S-CIFAR-10, CIL · Last accuracy | 64.11 -> 62.04 | Hurt | Do Your Best and Get Enough Rest for Continual Learning (CVPR 2025 · 1 citation) |
| Estimate task similarity using 1% of training data | CIFAR-10 · average test accuracy | Helped | Optimal Task Order for Continual Learning of Multiple Tasks (ICML 2025 · 1 citation) | |
| Use fixed q = s = 20% rather than tuning q and s | CIFAR-10-5, 10k buffer · mean test accuracy | 89.17 -> 86.15 | Hurt | Predicting the Susceptibility of Examples to Catastrophic Forgetting (ICML 2025 · 1 citation) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| TRGP | Split CIFAR-100 | ACC (%) | 74.46 – 80.5 | 4 papers, e.g. AAAI 2023 and CVPR 2023 |
| GPM | Split miniImageNet | BWT (%) | -0.7 – 0.3 | 3 papers, e.g. NeurIPS 2022 and CVPR 2023 |
| HAT | 20-Split mini-ImageNet | BWT (%) | -0.04 – -0.03 | 3 papers, e.g. CVPR 2021 and ICLR 2021 |
| GPM | Split CIFAR-100 | BWT | -0.9 – 0 | 2 papers, e.g. NeurIPS 2022 and AAAI 2023 |
What reviewers will expect
Datasets and baselines named in the 35 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-100 · 13 of 35 papers
- CIFAR-10 · 8 of 35 papers
- TinyImageNet · 6 of 35 papers
- MNIST · 5 of 35 papers
- DomainNet · 3 of 35 papers
- ImageNet100 · 3 of 35 papers
- Fashion-MNIST · 2 of 35 papers
- Split-CIFAR100 · 2 of 35 papers
Baselines
- EWC · 9 of 35 papers
- LwF · 7 of 35 papers
- DER · 7 of 35 papers
- ER · 6 of 35 papers
- GEM · 3 of 35 papers
- iCaRL · 3 of 35 papers
- L2P · 3 of 35 papers
- DualPrompt · 3 of 35 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 91 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 4 papers
- Huazhong University of Science and Technology · 3 papers
- Chinese Academy of Sciences · 3 papers
- Zhejiang University · 3 papers
- Eindhoven University of Technology · 3 papers
- Advanced Research Lab · 3 papers
- NavInfo Europe B.V · 3 papers
- ETHZ - ETH Zurich · 3 papers
- Amazon · 3 papers
- National University of Singapore · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Unified and unsupervised continual frameworks (120 papers)
- Gradient projection and plasticity preservation (80 papers)
- Dynamic expansion and isolation (70 papers)
- Plasticity-preserving continual reinforcement learning (43 papers)
Elsewhere in the topic
- Memory-based and Bayesian online learning (56 papers)
- Plug-and-play online training enhancements (39 papers)
- Replay buffer construction and update (33 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).