Research map / Continual learning / Online and replay-based
Improving Plasticity in Online Continual Learning via Collaborative Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CCL-DC, a collaborative (two peer learners) plus entropy-regularizing distillation-chain scheme that boosts plasticity and final accuracy when added to six online class-incremental CL methods.
Problem
Online continual learning research focuses on catastrophic forgetting (stability) while neglecting the model's ability to acquire new knowledge (plasticity), which limits final accuracy when data is seen only once.
Approach
Two peer continual learners train simultaneously with mutual KL distillation (CCL), plus a distillation chain over augmentations of increasing difficulty that distills from harder to easier views as a learned entropy regularizer (DC); the scheme is plug-in to existing replay-based online CL methods. A new metric, Relative Forgetting, and an analytic link AA ≳ LA×(1−RF) justify focusing on plastici
Key results
- Applying CCL-DC improves ER on CIFAR-100 (M=2k) from 31.89% to 44.45% average accuracy, and ER-ACE from 34.21% to 45.14%.
- On Tiny-ImageNet with M=2k, CCL-DC lifts ER from 10.82% to 16.56% average accuracy, a roughly 53% relative gain.
- Ablation on CIFAR-100 shows CCL alone and DC each add gains, and the harder-to-easier distillation direction outperforms easy-to-hard and same-difficulty schemes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CCL-DC to OnPro | OnPro · CIFAR-10, M=500 | average accuracy: 70.47±2.12 -> 74.49±2.14 | Helped |
| Add CCL-DC to ER-ACE | ER-ACE · CIFAR-100, M=2000 | average accuracy: 34.21±1.53 -> 45.14±1.00 | Helped |
| Add CCL-DC to OCM | OCM · Tiny-ImageNet, M=2000 | average accuracy: 18.36±0.95 -> 23.36±1.18 | Helped |
| Add CCL-DC to OnPro | OnPro · ImageNet-100, M=5000 | average accuracy: 38.75±1.03 -> 47.93±1.26 | Helped |
| Add multiview classification training without CCL-DC (ablation) | ER · CIFAR-100, M=2000 | average accuracy: 31.89±1.45 -> 38.18±1.46 | Helped |
| Add CCL without multiview learning or DC (ablation) | ER · CIFAR-100, M=2000 | average accuracy: 31.89±1.45 -> 41.05±1.21 | Helped |
| Add CCL-DC (ablation) | ER · CIFAR-100, M=2000 | average accuracy: 31.89±1.45 -> 44.45±1.04 | Helped |
| Add multiview classification training without CCL-DC (ablation) | ER-ACE · CIFAR-100, M=2000 | average accuracy: 34.21±1.53 -> 38.61±1.48 | Helped |
| Add CCL without multiview learning or DC (ablation) | ER-ACE · CIFAR-100, M=2000 | average accuracy: 34.21±1.53 -> 40.90±1.08 | Helped |
| Reverse DC distillation direction to easy-to-hard (ablation) | ER + CCL-DC · CIFAR-100, M=2000 | average accuracy: 44.45±1.04 -> 40.95±0.97 | Hurt |
| Use same-difficulty DC distillation (ablation) | ER + CCL-DC · CIFAR-100, M=2000 | average accuracy: 44.45±1.04 -> 43.64±1.09 | Hurt |
| Apply easy-to-hard self-distillation chain within one model (ablation) | ER · CIFAR-100, M=2000 | average accuracy: 31.89±1.45 -> 35.00±1.31 | Helped |
Surprising findings
- Distilling from an untrained, randomly initialized network already improves ER on CIFAR-100 (e.g., 31.89% to 34.84% at M=2k), suggesting overconfidence harms online CL.
- ER-ACE's plasticity is unexpectedly low on Tiny-ImageNet and its stability is compromised when CCL-DC is added, which the authors flag as an anomaly.
Limitations
- Two peer networks plus augmentation chain roughly double training compute relative to the baseline.
- Gains are reduced or absent for some settings, e.g., ER-ACE stability worsens and some RF values are not improved.
- Hyperparameters are tuned on CIFAR-100 with M=2k and transferred to other settings.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-100
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-100
Metrics: average accuracy learning accuracy (plasticity) relative forgetting (stability)
Baselines: ER DER++ ER-ACE OCM GSA OnPro
Builds on: Codistillation Deep Mutual Learning Experience Replay DER++ OnPro
Closest papers by meaning
- A Unified and General Framework for Continual Learning (ICLR 2024 · similarity 0.90)
- BECAME: Bayesian Continual Learning with Adaptive Model Merging (ICML 2025 · similarity 0.89)
- STAR: Stability-Inducing Weight Perturbation for Continual Learning (ICLR 2025 · similarity 0.89)
- Online Task-Free Continual Learning via Dynamic Expansionable Memory Distribution (CVPR 2025 · similarity 0.88)
- Interactive Continual Learning: Fast and Slow Thinking (CVPR 2024 · similarity 0.88)
- Do Your Best and Get Enough Rest for Continual Learning (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.