AI Paper Atlas

Research map / Continual learning / Online and replay-based

From Generic to Dedicated: A Novel Optimizer for Online Continual Learning

Yongyi Wu, Zheng Wang, Sen Lin

NeurIPS 2026 · Continual learning › Online and replay-based · approach: Plug-and-play online training enhancements · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a dedicated optimizer for online continual learning, motivated by a claimed link between the optimizer and forgetting, rather than using generic SGD or AdamW.

Problem

Online continual learning requires learning from non-stationary streams in a single pass, where models are highly susceptible to catastrophic forgetting; existing work largely overlooks the optimizer, which typically defaults to SGD or AdamW.

Approach

Reveals a link between the optimizer and catastrophic forgetting in OCL and designs an optimizer dedicated to the OCL setting instead of generic optimizers. Abstract is truncated, so the specific mechanism is not stated.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace AdamW with SPIN-AdamW as lower-level optimizer11 OCL methods · Split CIFAR-100 (M=2k)final average accuracy (%): 36.23 -> 38.49Improved
replace SGD with SPIN-SGD11 OCL methods · Split CIFAR-100 (M=2k)final average accuracy (%): 32.66 -> 36.54Improved
replace AdamW with SPIN-AdamW11 OCL methods · Split CIFAR-100 (M=2k)averaged anytime accuracy (%): 45.10 -> 49.75Improved
replace SGD with SPINDER++ on ImageNet-1K · ImageNet-1Kfinal accuracy (%): 14.19 -> 17.51Improved
use SPIN instead of SGD with EMA momentum (SGD-M)ER · CLEAR100 (M=200)accuracy (%): 36.01 -> 37.99Improved
set alpha=0 in SPIN (ablation)SPIN-AdamW with GSA · Split CIFAR-100 (M=1k)final accuracy (%): 32.20 -> 1.04Hurt
set beta=1 (gradients evaluated only on stability sequence) (ablation)SPIN-AdamW with GSA · Split CIFAR-100 (M=1k)final accuracy (%): 32.20 -> 22.30Hurt
use interpolated gradient evaluation (beta>0) instead of pure EMA (beta=0) (ablation)SPIN-AdamW with GSA · Split CIFAR-100 (M=1k)averaged anytime accuracy (%): 40.28 -> 42.39Improved
use parameter-space interpolation instead of gradient-space averaging (ablation)SPIN · Split CIFAR-100task-wise AAA and training timeImproved
replace SGD with SPIN-SGD on relative forgetting11 OCL methods · Split CIFAR-100 (M=2k)relative forgetting (%): 31.94 -> 29.99Mixed
SPIN with 15k buffer versus SGD with 30k buffer (ablation)GSA · Split CIFAR-100final accuracy (%): 46.93 -> 48.46Improved
use SPIN-SGD instead of SGDGSA · Split CIFAR-100runtime per step (ms): 1.04 -> 1.13Hurt

Experimental setup

Baselines: SGD AdamW

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy