Research map / Continual learning / Online and replay-based
From Generic to Dedicated: A Novel Optimizer for Online Continual Learning
Yongyi Wu, Zheng Wang, Sen Lin
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a dedicated optimizer for online continual learning, motivated by a claimed link between the optimizer and forgetting, rather than using generic SGD or AdamW.
Problem
Online continual learning requires learning from non-stationary streams in a single pass, where models are highly susceptible to catastrophic forgetting; existing work largely overlooks the optimizer, which typically defaults to SGD or AdamW.
Approach
Reveals a link between the optimizer and catastrophic forgetting in OCL and designs an optimizer dedicated to the OCL setting instead of generic optimizers. Abstract is truncated, so the specific mechanism is not stated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace AdamW with SPIN-AdamW as lower-level optimizer | 11 OCL methods · Split CIFAR-100 (M=2k) | final average accuracy (%): 36.23 -> 38.49 | Improved |
| replace SGD with SPIN-SGD | 11 OCL methods · Split CIFAR-100 (M=2k) | final average accuracy (%): 32.66 -> 36.54 | Improved |
| replace AdamW with SPIN-AdamW | 11 OCL methods · Split CIFAR-100 (M=2k) | averaged anytime accuracy (%): 45.10 -> 49.75 | Improved |
| replace SGD with SPIN | DER++ on ImageNet-1K · ImageNet-1K | final accuracy (%): 14.19 -> 17.51 | Improved |
| use SPIN instead of SGD with EMA momentum (SGD-M) | ER · CLEAR100 (M=200) | accuracy (%): 36.01 -> 37.99 | Improved |
| set alpha=0 in SPIN (ablation) | SPIN-AdamW with GSA · Split CIFAR-100 (M=1k) | final accuracy (%): 32.20 -> 1.04 | Hurt |
| set beta=1 (gradients evaluated only on stability sequence) (ablation) | SPIN-AdamW with GSA · Split CIFAR-100 (M=1k) | final accuracy (%): 32.20 -> 22.30 | Hurt |
| use interpolated gradient evaluation (beta>0) instead of pure EMA (beta=0) (ablation) | SPIN-AdamW with GSA · Split CIFAR-100 (M=1k) | averaged anytime accuracy (%): 40.28 -> 42.39 | Improved |
| use parameter-space interpolation instead of gradient-space averaging (ablation) | SPIN · Split CIFAR-100 | task-wise AAA and training time | Improved |
| replace SGD with SPIN-SGD on relative forgetting | 11 OCL methods · Split CIFAR-100 (M=2k) | relative forgetting (%): 31.94 -> 29.99 | Mixed |
| SPIN with 15k buffer versus SGD with 30k buffer (ablation) | GSA · Split CIFAR-100 | final accuracy (%): 46.93 -> 48.46 | Improved |
| use SPIN-SGD instead of SGD | GSA · Split CIFAR-100 | runtime per step (ms): 1.04 -> 1.13 | Hurt |
Experimental setup
Baselines: SGD AdamW
Closest papers by meaning
- Online Curvature-Aware Replay: Leveraging $\mathbf{2^{nd}}$ Order Information for Online Continual Learning (ICML 2025 · similarity 0.90)
- Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretization (CVPR 2025 · similarity 0.90)
- Gradient-Guided Epsilon Constraint Method for Online Continual Learning (NeurIPS 2025 · similarity 0.90)
- Ferret: An Efficient Online Continual Learning Framework under Varying Memory Constraints (CVPR 2025 · similarity 0.89)
- A Statistical Theory of Regularization-Based Continual Learning (ICML 2024 · similarity 0.88)
- Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
