Research map / Continual learning / Online and replay-based
Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits online continual learning by highlighting model throughput, model ignorance and myopia, and proposes NsCE (non-sparse classifier evolution) with sparsity and maximum-separation regularizers plus targeted replay, using pre-trained ViT backbones.
Problem
Online continual learning methods mostly target catastrophic forgetting while ignoring model throughput under high-speed streams; models also fail to learn effective features in single-pass training and become myopic with sparse classifiers.
Approach
Defines model throughput as a first-class OCL metric, links it to a PAC-Bayes bound, and proposes NsCE: an L_s sparsity regularizer, a maximum-separation (ETF-style) loss, and targeted binary replay driven by a confusion matrix, with replay accessed only every few iterations.
Key results
- NsCE achieves A_AUC of 89.9 on CIFAR10 (M=0.1k, Freq=1/100) versus 83.8 for the best baseline SCR listed in the table.
- On CIFAR100 (M=0.5k, Freq=1/100) NsCE reaches A_AUC 74.1 vs 62.9 for the strongest baseline shown.
- On ImageNet (M=10k, Freq=1/500) NsCE reaches A_AUC 61.6 vs 57.1 for iCaRL; ablation shows targeted ER is the largest single gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add non-sparse regularization L_s (ablation) | ER with vanilla cross-entropy · CIFAR-10 (M=0.1k, Freq=1/100) | A_AUC: 82.6{$\pm$0.5} -> 84.5{$\pm$1.3} | Helped |
| Add maximum separation L_p alongside L_s (ablation) | ER with vanilla cross-entropy · CIFAR-10 (M=0.1k, Freq=1/100) | A_AUC: 84.5{$\pm$1.3} -> 86.2{$\pm$0.8} | Helped |
| Use targeted ER instead of standard ER (ablation) | vanilla cross-entropy model · CIFAR-10 (M=0.1k, Freq=1/100) | A_AUC: 82.6{$\pm$0.5} -> 87.2{$\pm$0.9} | Helped |
| Replace NsCE with NsCE Lite (ablation) | NsCE · CIFAR-10 (M=0.1k, Freq=1/100) | A_AUC: 89.9$\pm$0.4 -> 90.7$\pm$0.6 | Helped |
| Replace NsCE with NsCE Lite (ablation) | NsCE · CIFAR-100 (M=0.5k, Freq=1/100) | A_AUC: 74.1$\pm$0.7 -> 72.9$\pm$0.9 | Hurt |
| Replace NsCE with NsCE Lite (ablation) | NsCE · EuroSat (M=0.1k, Freq=1/100) | A_AUC: 75.7$\pm$0.4 -> 76.1$\pm$0.5 | Helped |
| Replace NsCE with NsCE Lite (ablation) | NsCE · CLEAR-10 (M=0.1k, Freq=1/100) | A_AUC: 89.2$\pm$0.7 -> 88.0$\pm$0.6 | Hurt |
| Replace NsCE with NsCE Lite (ablation) | NsCE · CLEAR-100 (M=1k, Freq=1/100) | A_AUC: 84.3$\pm$0.4 -> 82.9$\pm$1.2 | Hurt |
| Replace NsCE with NsCE Lite (ablation) | NsCE · ImageNet (M=10k, Freq=1/500) | A_AUC: 61.6$\pm$0.7 -> 60.9$\pm$0.8 | Hurt |
| Use NsCE instead of the strongest listed baseline | online continual learning · CIFAR-10 (M=0.1k, Freq=1/100) | A_AUC: 83.8{$\pm0.2$} -> 89.9{$\pm$0.4} | Helped |
| Use NsCE instead of the strongest listed baseline | online continual learning · CIFAR-100 (M=0.5k, Freq=1/100) | A_AUC: 62.9{$\pm$0.6} -> 74.1{$\pm$0.7} | Helped |
| Use NsCE instead of the strongest listed baseline | online continual learning · EuroSat (M=0.1k, Freq=1/100) | A_AUC: 61.3{$\pm$0.9} -> 75.7{$\pm$0.4} | Helped |
Surprising findings
- Replay-based strategies with extra computation (ER, contrastive replay, distillation) were no more effective than simply training CE multiple times (CE++) at equal throughput.
- Training from scratch on CIFAR100 keeps average accuracy below 10% even without inter-task interference, which the authors attribute to model ignorance.
Limitations
- Theoretical bound only covers per-task generalization risk and cannot represent a global risk for disjoint task streams.
- Relies on pre-trained initialization; the authors note the choice of pre-trained model is not the paper's focus and effects vary.
Experimental setup
Datasets: CIFAR10 CIFAR100 EuroSat CLEAR10 CLEAR100 ImageNet
Benchmarks: CIFAR10 CIFAR100 EuroSat CLEAR10 CLEAR100 ImageNet
Metrics: A_AUC (area under accuracy curve) last accuracy
Baselines: iCaRL EWC DER++ PASS MC-SGD w/ SAM AGEM ER MIR ASER SCR DVC OCM
Builds on: neural collapse experience replay PAC-Bayes
Closest papers by meaning
- The Importance of Being Lazy: Scaling Limits of Continual Learning (ICML 2025 · similarity 0.88)
- Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretization (CVPR 2025 · similarity 0.88)
- Federated Continual Learning Goes Online: Uncertainty-Aware Memory Management for Vision Tasks and Beyond (ICLR 2025 · similarity 0.88)
- Kalman Filter for Online Classification of Non-Stationary Data (ICLR 2024 · similarity 0.88)
- Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers (ICML 2025 · similarity 0.88)
- AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projection (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.