Research map / Continual learning / Online and replay-based
Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MOSE, which splits a ResNet18 into stacked latent experts trained with multi-level supervision (CE + supervised contrastive) and distilled back to the final network via reverse self-distillation, for online continual learning.
Problem
Online continual learning (one-pass data streams with replay buffers) suffers from an overfitting-underfitting dilemma: new-task samples are underfit while buffered old samples are overfit.
Approach
Attach alignment modules and output heads to each ResNet block so every depth acts as a sub-expert trained with CE and supervised contrastive losses; then distill the ensemble of shallower experts into the final network (reverse self-distillation), so inference uses only the original network.
Key results
- MOSE achieves 38.7% ACC on Split CIFAR-100 at M=1k versus 31.4% for GSA (the prior best), with MOE-MOSE reaching a 7.3% gain over the best baseline on Split CIFAR-100 and 6.1% on Split Tiny-ImageNet.
- Ablation on Split CIFAR-100 (M=5k): baseline 36.89%, adding MLS 45.00%, and full MOSE with RSD 54.57%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MOSE instead of GSA | ResNet18 online continual learner · Split CIFAR-100, M=1k | ACC (%): 31.4±0.2 -> 35.1±0.4 | Helped |
| Use MOSE instead of GSA | ResNet18 online continual learner · Split CIFAR-100, M=2k | ACC (%): 39.7±0.6 -> 45.1±0.3 | Helped |
| Use MOSE instead of GSA | ResNet18 online continual learner · Split CIFAR-100, M=5k | ACC (%): 49.7±0.2 -> 54.8±0.4 | Helped |
| Use MOSE instead of GSA | ResNet18 online continual learner · Split Tiny-ImageNet, M=2k | ACC (%): 18.4±0.4 -> 19.4±0.5 | Helped |
| Use MOSE instead of GSA | ResNet18 online continual learner · Split Tiny-ImageNet, M=4k | ACC (%): 26.0±0.2 -> 28.0±0.8 | Helped |
| Use MOSE instead of GSA | ResNet18 online continual learner · Split Tiny-ImageNet, M=10k | ACC (%): 33.2±0.4 -> 38.7±0.4 | Helped |
| Add multi-level supervision with CE and SCL (ablation) | MOSE · Split CIFAR-100, M=5k | ACC (%): 45.00 -> 52.29 | Helped |
| Add multi-level supervision with CE (ablation) | MOSE · Split CIFAR-100, M=5k | ACC (%): 36.89 -> 40.24 | Helped |
| Add multi-level supervision with SCL (ablation) | MOSE · Split CIFAR-100, M=5k | ACC (%): 39.36 -> 46.21 | Helped |
| Add reverse self-distillation (ablation) | MOSE with multi-level supervision · Split CIFAR-100, M=5k | ACC (%): 52.29 -> 54.57 | Helped |
| Add reverse self-distillation to the final expert (ablation) | MOSE experts · Split CIFAR-100, M=5000 | Average accuracy: 52.63 -> 54.57 | Helped |
| Average output logits from all experts instead of using the final expert | MOSE · Split CIFAR-100, M=5k | ACC (%): 54.8±0.4 -> 55.6±0.4 | Helped |
Surprising findings
- Averaging logits of all experts (MOE) outperforms even the oracle best single expert per task, indicating complementary expert skills.
Limitations
- Evaluated only on image classification benchmarks with ResNet18 backbone.
- MOE variant incurs extra computation and storage costs.
Experimental setup
Datasets: Split CIFAR-100 Split Tiny-ImageNet
Benchmarks: Split CIFAR-100 Split Tiny-ImageNet
Metrics: average accuracy average forgetting buffer overfitting factor
Baselines: AGEM ER MIR GSS ASER ER-AML GDumb SCR OCM OnPro GSA DER++
Builds on: Experience Replay Supervised Contrastive Replay OCM Self-distillation SimCLR
Closest papers by meaning
- Rethinking Momentum Knowledge Distillation in Online Continual Learning (ICML 2024 · similarity 0.88)
- Theory on Mixture-of-Experts in Continual Learning (ICLR 2025 · similarity 0.88)
- Improving Plasticity in Online Continual Learning via Collaborative Learning (CVPR 2024 · similarity 0.88)
- Layerwise Proximal Replay: A Proximal Point Method for Online Continual Learning (ICML 2024 · similarity 0.87)
- Mitigating Catastrophic Forgetting in Online Continual Learning by Modeling Previous Task Interrelations via Pareto Optimization (ICML 2024 · similarity 0.86)
- Mixture of Experts Meets Prompt-Based Continual Learning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.