Research map / Continual learning / Online and replay-based
A Unified and General Framework for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a unified Bregman-divergence framework that recovers VCL, NCL, EWC, CPR, ER and DER as special cases, and a "refresh learning" unlearn-then-relearn plug-in derived from a Fokker-Planck/Langevin view that improves continual learning on CIFAR and Tiny-ImageNet benchmarks.
Problem
Continual learning methods (regularization, Bayesian, replay) lack a common mathematical framework, and they tend to over-memorize outdated knowledge, hurting learning of new tasks and generalization under catastrophic forgetting.
Approach
A general CL objective built from Bregman divergences in output and weight space subsumes several CL methods; refresh learning adds a noisy unlearning step (Langevin-style gradient flow on the current loss, preconditioned by the inverse FIM) before the usual relearning step, which approximately minimizes an FIM-weighted gradient norm.
Key results
- DER++ with refresh reaches 77.71% Task-IL on CIFAR-100 vs 75.64% for DER++ alone, and 74.42% vs 72.70% Class-IL on CIFAR-10.
- ER+refresh improves Class-IL on CIFAR-100 from 20.98% to 22.23% and Tiny-ImageNet Class-IL from 9.99% to 11.09% (memory size 500).
- Refresh adds about 0.81x running time overhead over DER++ on CIFAR-100 (8.4s vs 15.2s per epoch, as reported).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add refresh to DER++ | DER++ · CIFAR-10, CIFAR-100, and Tiny-ImageNet, Class-IL and Task-IL; memory 500 | overall accuracy: 72.70± 1.36 -> 74.42 ± 0.82; 93.88±0.50 -> 94.64 ± 0.38; 36.37±0.85 -> 38.49 ± 0.76; 75.64±0.60 -> 77.71 ± 0.85; 19.38±1 | Helped |
| Add refresh to ER | ER · CIFAR-10, CIFAR-100, and Tiny-ImageNet, Class-IL and Task-IL; memory 500 | overall accuracy: 57.74±0.27 -> 61.86 ± 1.35; 93.61±0.27 -> 94.15 ± 0.46; 20.98±0.35 -> 22.23 ± 0.73; 73.37±0.43 -> 75.45 ± 0.67; 9.99±0.2 | Helped |
| Add refresh to oEWC | oEWC · CIFAR-10, CIFAR-100, and Tiny-ImageNet, Class-IL and Task-IL; memory 500 | overall accuracy: 19.49± 0.12 -> 20.37 ± 0.65; 64.31 ± 4.31 -> 66.89 ± 2.57; 8.24± 0.21 -> 8.78 ± 0.42; 21.2± 2.08 -> 23.31 ± 1.87; 7.42± | Mixed |
| Add refresh to CPR(EWC) | CPR(EWC) · CIFAR-10, CIFAR-100, and Tiny-ImageNet, Class-IL and Task-IL; memory 500 | overall accuracy: 19.61± 3.67 -> 20.53 ± 2.42; 65.23 ± 3.87 -> 67.36 ± 3.68; 8.42± 0.37 -> 9.06 ± 0.58; 21.43± 2.57 -> 22.90 ± 1.71; 7.67± | Helped |
| Add refresh to DER++ | DER++ · P-MNIST and R-MNIST Domain-IL; memory 500 | overall accuracy: 88.21±0.39 -> 88.93 ± 0.58; 92.77±1.05 -> 93.28 ± 0.75 | Helped |
| Add refresh to DER++ | DER++ · CIFAR-100 and Tiny-ImageNet, Class-IL and Task-IL; memory 2000 | overall accuracy: 50.72± 0.71 -> 52.81 ± 0.80; 82.43± 0.38 -> 84.05 ± 0.77; 24.21± 1.09 -> 27.37 ± 1.53; 62.22± 0.87 -> 64.31 ± 0.98 | Helped |
| Set unlearning rate gamma to 0.02, 0.03, or 0.04 (ablation) | Refresh learning · CIFAR-100, CIFAR-10, and Tiny-ImageNet Task-IL | accuracy: CIFAR100: 77.23 ± 0.97; 77.71 ± 0.85; 77.08 ± 0.90. CIFAR10: 94.27 ± 0.42; 94.64 ± 0.38; 94.82 ± 0.51. Tiny-ImageNet: 53 | Mixed |
| Use 2 rather than 1 unlearning step (ablation) | Refresh learning · CIFAR-100, CIFAR-10, and Tiny-ImageNet Task-IL | accuracy: CIFAR100: 77.71 ± 0.85 -> 77.76 ± 0.82; CIFAR10: 94.64 ± 0.38 -> 94.73 ± 0.43; Tiny-ImageNet: 54.06 ± 0.79 -> 54.17 ± 0. | No clear effect |
| Use 3 rather than 1 unlearning step (ablation) | Refresh learning · CIFAR-100, CIFAR-10, and Tiny-ImageNet Task-IL | accuracy: CIFAR100: 77.71 ± 0.85 -> 75.93 ± 1.06; CIFAR10: 94.64 ± 0.38 -> 93.50 ± 0.57; Tiny-ImageNet: 54.06 ± 0.79 -> 52.29 ± 0. | Hurt |
Limitations
- Gains are modest (often 1-2 points) and the reported standard deviations overlap in many cases.
- The theoretical link relies on several Taylor approximations and the claim that the gradient of the regularizer is near zero at the stationary point.
- Experiments are limited to small image benchmarks with ResNet18 and do not test large models.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet P-MNIST R-MNIST
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet P-MNIST R-MNIST
Metrics: overall accuracy (ACC) backward transfer (BWT)
Baselines: oEWC SI LwF CPR GPM NCL HAT ER A-GEM GSS DER++ HAL
Builds on: Bregman divergence elastic weight consolidation variational continual learning dark experience replay Fokker-Planck equation
Closest papers by meaning
- Mitigating Catastrophic Forgetting in Online Continual Learning by Modeling Previous Task Interrelations via Pareto Optimization (ICML 2024 · similarity 0.91)
- Do Your Best and Get Enough Rest for Continual Learning (CVPR 2025 · similarity 0.91)
- Predicting the Susceptibility of Examples to Catastrophic Forgetting (ICML 2025 · similarity 0.91)
- Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.91)
- BECAME: Bayesian Continual Learning with Adaptive Model Merging (ICML 2025 · similarity 0.90)
- Online Task-Free Continual Learning via Dynamic Expansionable Memory Distribution (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.