Research map / Continual learning / Online and replay-based
Rethinking Momentum Knowledge Distillation in Online Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies momentum knowledge distillation (an EMA-of-student teacher with multiview logit distillation) to replay-based online continual learning methods, improving final accuracy across CIFAR10/100, Tiny-ImageNet and ImageNet100.
Problem
Knowledge distillation is underused in online continual learning (OCL) because frozen snapshot teachers suffer from poor teacher quality, growing teacher count, and unknown task boundaries in single-pass class-incremental streams.
Approach
Use an evolving teacher given by an exponential moving average of the student's weights, distilled in logit space with multiview (raw and augmented) KL loss, plus averaging student and teacher weights at inference; this is a plug-in loss applicable to many OCL baselines without task boundaries.
Key results
- On ImageNet100 (M=10k, clear setting), GSA + ours reaches 52.41% vs 40.18% for GSA alone, a gain of over 10 points.
- ER + ours improves ER on CIFAR100 M=5k from 38.05% to 52.10% in the clear setting.
- ER + ours achieves positive backward transfer (+8.15%) on CIFAR100 M=5k where ER has -16.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MKD to ER | ER · CIFAR100 clear, M=5000 | final average accuracy (%): 38.05{ ±1.08} -> 52.10{ ±0.5} | Helped |
| Add MKD to OCM | OCM · CIFAR100 clear, M=5000 | final average accuracy (%): 41.87{ ±1.52} -> 51.24{ ±0.81} | Helped |
| Add MKD to GSA | GSA · ImageNet100 clear, M=10000 | final average accuracy (%): 40.18{ ±0.93} -> 52.41{ ±0.59} | Helped |
| Add MKD to OCM | OCM · CIFAR100 blurry, M=5000 | final average accuracy (%): 38.14{ ±1.11} -> 51.41{ ±0.71} | Helped |
| Use student weights alone for inference rather than averaging student and teacher weights (ablation) | ER + ours · CIFAR100 clear, M=1000 | final average accuracy (%): 38.5{ ±0.5} -> 37.7{ ±0.7} | Hurt |
| Use teacher weights alone for inference rather than averaging student and teacher weights (ablation) | ER + ours · CIFAR100 clear, M=1000 | final average accuracy (%): 38.5{ ±0.5} -> 37.2{ ±0.7} | Hurt |
| Use single-view distillation instead of multiview distillation (ablation) | ER + ours · CIFAR100 clear, M=1000 | final average accuracy (%): 38.5{ ±0.5} -> 34.8{ ±0.6} | Hurt |
| Add MKD to ER | ER · CIFAR100 clear, M=5000 | backward transfer (%): -16.7{ ±1.2 } -> +\textbf{8.15{ ±0.8}} | Helped |
Surprising findings
- Applying NCM classification on top of MKD-trained models degrades accuracy, suggesting MKD in logit space already neutralizes last-layer bias.
- Teacher alone performs worse than the student, yet averaging teacher and student weights for inference gives the best accuracy.
Limitations
- Requires tuning the momentum alpha and a teacher-dependent weight lambda; grid search is used to pick them.
- For OCM, the method does not reduce feature drift, and the blurry-setting task boundaries had to be inferred with heuristic rules.
- GSA could not be run in the blurry setting because it needs exact class-task relationships.
Experimental setup
Datasets: CIFAR10 CIFAR100 Tiny-ImageNet ImageNet100
Benchmarks: CIFAR10 CIFAR100 Tiny-ImageNet ImageNet100
Metrics: final average accuracy standard deviation backward transfer feature drift t-SNE
Baselines: ER DER++ ER-ACE DVC OCM GSA PCR Temp. Ens. SDP
Builds on: Experience Replay DER++ Momentum Knowledge Distillation GSA OCM
Closest papers by meaning
- Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning (AAAI 2024 · similarity 0.89)
- Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation (CVPR 2024 · similarity 0.88)
- Improving Plasticity in Online Continual Learning via Collaborative Learning (CVPR 2024 · similarity 0.88)
- Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretization (CVPR 2025 · similarity 0.88)
- Enhancing Visual Continual Learning with Language-Guided Supervision (CVPR 2024 · similarity 0.87)
- Online Curvature-Aware Replay: Leveraging $\mathbf{2^{nd}}$ Order Information for Online Continual Learning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.