Research map / Continual learning / Online and replay-based
Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S6MOD, a plug-and-play selective state space branch with class-conditional mixture of discretization and contrastive discretization loss that boosts existing online continual learning methods on CIFAR-10/100 and Tiny-ImageNet.
Problem
Online continual learning requires learning generalizable, discriminative features incrementally from single-pass data streams; existing replay methods focus on memory retention and neglect model adaptability.
Approach
An extra selective SSM (S6) branch after the backbone whose discretization step is produced by a sparse mixture of discretization candidates, routed per class by prototype-based uncertainty (more candidates for confusable classes), plus a contrastive loss on the aggregated discretization; it is attached to existing OCL methods with an ETF classifier and KL supervision.
Key results
- S6MOD improves OnPro on CIFAR-100 (M=2k) from 33.33% to 37.57% average accuracy.
- OnPro-CCLDC with S6MOD reaches 79.51% on CIFAR-10 (M=1k), up from 78.64%.
- MOE-MOSE with S6MOD reaches 56.32% on CIFAR-100 (M=5k) and 40.11% on Tiny-ImageNet (M=10k).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add S6MOD to OnPro-CCLDC | OnPro-CCLDC · CIFAR-10, M=500 and M=1000 | Average accuracy (%): 74.49±2.14 -> 75.03±1.03; 78.64±1.42 -> 79.51±0.63 | Helped |
| Add S6MOD to MOE-MOSE | MOE-MOSE · CIFAR-100, M=1000, M=2000, and M=5000 | Average accuracy (%): 37.32±0.34 -> 37.76±0.51; 47.03±0.57 -> 47.25±0.43; 55.62±0.72 -> 56.32±0.53 | Helped |
| Add S6MOD to OCM-CCLDC | OCM-CCLDC · Tiny-ImageNet, M=2000, M=5000, and M=10000 | Average accuracy (%): 23.36±1.18 -> 23.68±1.02; 33.17±0.97 -> 33.59±0.72; 39.25±0.88 -> 39.53±1.02 | Helped |
| Add an SSM branch and LDiff (ablation) | OnPro · CIFAR-100, M=2000 | Average accuracy (%): 33.33±0.93 -> 36.61±1.04 | Helped |
| Add class-conditional routing to the branch with LDiff (ablation) | OnPro · CIFAR-100, M=2000 | Average accuracy (%): 36.61±1.04 -> 36.92±0.72 | Helped |
| Add class-conditional routing to the branch with LDiff (ablation) | OnPro · CIFAR-100, M=2000 | Average forgetting (%): 10.28±1.59 -> 8.85±0.81 | Helped |
| Add contrastive discretization loss to the branch with LDiff (ablation) | OnPro · CIFAR-100, M=2000 | Average accuracy (%): 36.61±1.04 -> 36.93±0.84 | Helped |
| Add contrastive discretization loss to the branch with LDiff (ablation) | OnPro · CIFAR-100, M=2000 | Average forgetting (%): 10.28±1.59 -> 9.70±1.42 | Helped |
| Use both class-conditional routing and LCont in the full S6MOD (ablation) | OnPro · CIFAR-100, M=2000 | Average accuracy (%): 36.61±1.04 -> 37.57±0.81 | Helped |
| Vary the LDiff weight α from 0.01 to 5 (ablation) | S6MOD on OnPro · CIFAR-100, M=2000 | Average accuracy (%): 36.80±1.17; 37.09±1.09; 37.55±1.08; 37.47±0.94; 37.57±0.81 | Helped |
| Vary the LCont weight β from 0.1 to 25 (ablation) | S6MOD on OnPro · CIFAR-100, M=2000 | Average accuracy (%): 36.50±1.24; 37.30±1.03; 37.40±0.33; 37.26±0.61; 37.57±0.81 | Helped |
| Increase the number of discretization patterns N (ablation) | S6MOD · CIFAR-100, M=2000 | New-task average accuracy | Helped |
Surprising findings
- Adding class-conditional routing or the contrastive discretization loss alone gives only about 0.3% gain; they help mainly when combined.
- In several settings the module increases average forgetting (e.g., OnPro on Tiny-ImageNet, OnPro-CCLDC/MOSE in some cases), which the authors attribute to stronger learning rather than instability.
Limitations
- Gains are often around 1% on CIFAR datasets and some settings show higher forgetting.
- Routing and contrastive loss individually give only marginal improvements.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet
Metrics: average accuracy average forgetting new-task accuracy
Baselines: ER OCM OnPro OCM-CCLDC OnPro-CCLDC MOSE MOE-MOSE
Builds on: Mamba (S6) VMamba (SS2D) ER OCM OnPro CCLDC MOSE
Closest papers by meaning
- Non-exemplar Online Class-Incremental Continual Learning via Dual-Prototype Self-Augment and Refinement (AAAI 2024 · similarity 0.88)
- Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learning (NeurIPS 2024 · similarity 0.88)
- Ferret: An Efficient Online Continual Learning Framework under Varying Memory Constraints (CVPR 2025 · similarity 0.88)
- Online Curvature-Aware Replay: Leveraging $\mathbf{2^{nd}}$ Order Information for Online Continual Learning (ICML 2025 · similarity 0.88)
- Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.88)
- Provable Contrastive Continual Learning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.