Research map / Continual learning / Online and replay-based
Gradient projection and plasticity preservation
Constrains updates to subspaces or preserves network trainability to limit forgetting.
80 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Online and replay-based, keyword cluster “weight · transfer · growing”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Few recent reports The tracked benchmarks have at most one report from 2025–26 that could beat the old record, too few to say whether they are still moving.
16 papers in 2023–24 and 19 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 9.8% to 9.6%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning (CVPR 2018 · 1,240 citations)
- 2019 · An Empirical Study of Example Forgetting during Deep Neural Network Learning (ICLR 2019 · 106 citations)
- 2020 · Conditional Channel Gated Networks for Task-Aware Continual Learning (CVPR 2020 · 177 citations)
- 2021 · Mixed-Privacy Forgetting in Deep Networks (CVPR 2021 · 123 citations)
- 2022 · DyTox: Transformers for Continual Learning With DYnamic TOken eXpansion (CVPR 2022 · 355 citations)
- 2023 · Sparse Coding in a Dual Memory System for Lifelong Learning (AAAI 2023 · 20 citations)
- 2024 · Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning (AAAI 2024 · 28 citations)
- 2025 · On Local Overfitting and Forgetting in Deep Neural Networks (AAAI 2025 · 2 citations)
- 2026 · HiCL: Hippocampal-Inspired Continual Learning (AAAI 2026 · 2 citations)
Earliest papers here
- PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning (CVPR 2018 · 1,240 citations)
- Measuring Catastrophic Forgetting in Neural Networks (AAAI 2018 · 574 citations)
- FearNet: Brain-Inspired Model for Incremental Learning (ICLR 2018 · 318 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
CIFAR-10 · accuracy (higher is better)
4 papers, 2022–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 95.91 | Fortuitous Forgetting in Connectionist Networks (ICLR 2022 · 5 citations) |
What has been tried
64 papers here report 683 measured changes (a component added, removed or swapped, compared with a baseline): helped 63%, no clear effect 10%, hurt 20%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add trace-based K-WTA instead of using a rate-based mask | splitCIFAR10 · validation accuracy | 76.88% -> 77.73% | Helped | Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning (AAAI 2024 · 28 citations) |
| Replace EM-derived connectivity with shuffled connectivity, removing its low-rank structure | Neuroscience tasks · effective learning laziness | No clear effect | How connectivity structure shapes rich and lazy learning in neural circuits (ICLR 2024 · 4 citations) | |
| Apply KF to the trained model | CIFAR-100 · test accuracy | 80.72 ± .53 -> 81.64 ± .27 | Helped | On Local Overfitting and Forgetting in Deep Neural Networks (AAAI 2025 · 2 citations) |
| Use spike trains rather than weighted firing rate for Hebbian learning | PMNIST · ACC (%) | 95.15 -> 95.23 | No clear effect | Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks (ICLR 2024 · 2 citations) |
| Use only the direct-gradient component (DG) in ACN training | CIFAR-10 · layer-wise compression | Hurt | Auto-Compressing Networks (NeurIPS 2025 · 0 citations) | |
| remove synaptic renormalization (only consolidation remains) | three EEG datasets, 30% source · performance | Hurt | SPICED: A Synaptic Homeostasis-Inspired Framework for Unsupervised Continual EEG Decoding (NeurIPS 2025 · 0 citations) | |
| Add SNR to attention-only L2 regularization | all tasks · loss | 0.1506 -> 0.1402 | Helped | Self-Normalized Resets for Plasticity in Continual Learning (ICLR 2025 · 0 citations) |
| Remove regularization | Continual learning on tiny-ImageNet, CIFAR10, and CIFAR100 · test accuracy | Hurt | Plastic Learning with Deep Fourier Features (ICLR 2025 · 0 citations) | |
| Replace the covariance regularizer with identity (L2) | Split-MNIST (SH), 40 points/task · accuracy | 89.22 ± 0.76 -> 87.53 ± 0.36 | Hurt | Function-space Parameterization of Neural Networks for Sequential Learning (ICLR 2024 · 0 citations) |
| Use dynamic α rather than fixed α | ImageNet-V1 · top-1 accuracy | 74.15 -> 74.53 | Helped | Forget Sharpness: Perturbed Forgetting of Model Biases Within SAM Dynamics (ICML 2024 · 0 citations) |
| Increase model size | CNN/Daily Mail cyclic fine-tuning · anticipatory recovery score | Helped | Reawakening knowledge: Anticipatory recovery from catastrophic interference via structured training (NeurIPS 2024 · 0 citations) | |
| Remove the invariance loss | CoraFull, Arxiv, Reddit · mean AP | Hurt | Disentangled Continual Graph Neural Architecture Search with Invariant Modular Supernet (ICML 2024) |
What reviewers will expect
Datasets and baselines named in the 10 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-100 · 5 of 10 papers
- MNIST · 4 of 10 papers
- CIFAR10 · 4 of 10 papers
- TinyImagenet · 3 of 10 papers
- EMNIST · 2 of 10 papers
Baselines
- L2 regularization · 3 of 10 papers
- L2 Init · 2 of 10 papers
- ReDo · 2 of 10 papers
- Continual Backprop · 2 of 10 papers
- Shrink and Perturb · 2 of 10 papers
- L2 regularization towards initialization · 2 of 10 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 35 papers; from OpenAlex and the papers' affiliations).
- Zhejiang University · 2 papers
- Mila · 2 papers
- Google DeepMind · 2 papers
- Korea Advanced Institute of Science and Technology · 2 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Theoretical analysis of forgetting (141 papers)
- Unified and unsupervised continual frameworks (120 papers)
- Dynamic expansion and isolation (70 papers)
- Plasticity-preserving continual reinforcement learning (43 papers)
Elsewhere in the topic
- Memory-based and Bayesian online learning (56 papers)
- Plug-and-play online training enhancements (39 papers)
- Replay buffer construction and update (33 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).