Research map / Continual learning / Online and replay-based
Theory on Mixture-of-Experts in Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Provides the first theoretical analysis of sparsely-gated mixture-of-experts in continual learning using overparameterized linear regression, showing expert diversification, the need to terminate gating updates, and explicit forgetting/generalization error expressions.
Problem
Lack of theoretical understanding of how Mixture-of-Experts affects catastrophic forgetting and generalization in continual learning when tasks arrive sequentially.
Approach
Models MoE in CL as overparameterized linear regression with a softmax top-1 router with noise perturbation, a locality loss plus auxiliary load-balancing loss for gating updates, and an early termination rule for gating updates; derives closed-form expected forgetting and overall generalization error and compares with a single expert.
Key results
- On CIFAR-100 task sequence, M=10 with early termination reaches 32.9% average incremental accuracy vs 17.1% for a single expert and 9.1% without early termination.
- On Tiny ImageNet, M=8 with early termination achieves 37.6% vs 17.4% for M=1 and 10.5% without termination.
- Theory: MoE with early termination drives forgetting and generalization error to near zero after O(M) rounds, while adding experts beyond the number of clusters delays convergence without improving performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoE with M=8 instead of a single expert | continual learning model · CIFAR-100 | test accuracy (%): 17.1 -> 25.5 | Helped |
| Use MoE with M=10 instead of a single expert | continual learning model · CIFAR-100 | test accuracy (%): 17.1 -> 32.9 | Helped |
| Use MoE with M=8 instead of a single expert | continual learning model · Tiny ImageNet | test accuracy (%): 17.4 -> 37.6 | Helped |
| Use MoE with M=10 instead of a single expert | continual learning model · Tiny ImageNet | test accuracy (%): 17.4 -> 28.3 | Helped |
| Add early termination of gating-network updates (ablation) | MoE with M=8 · CIFAR-100 | test accuracy (%): 10.3 -> 25.5 | Helped |
| Add early termination of gating-network updates (ablation) | MoE with M=10 · CIFAR-100 | test accuracy (%): 9.1 -> 32.9 | Helped |
| Add early termination of gating-network updates (ablation) | MoE with M=8 · Tiny ImageNet | test accuracy (%): 10.5 -> 37.6 | Helped |
| Add early termination of gating-network updates (ablation) | MoE with M=10 · Tiny ImageNet | test accuracy (%): 10.3 -> 28.3 | Helped |
| Increase expert count from M=8 to M=10 | MoE · CIFAR-100 | test accuracy (%): 25.5 -> 32.9 | Helped |
| Increase expert count from M=8 to M=10 | MoE · Tiny ImageNet | test accuracy (%): 37.6 -> 28.3 | Hurt |
| Add more experts | MoE in continual learning · continual learning (theoretical analysis) | learning performance | No clear effect |
Surprising findings
- Continuing to update the gating network in CL eventually collapses expert gate gaps and causes wrong routing, so early termination of gating updates is necessary.
- Adding more experts than task clusters delays convergence and does not improve learning performance.
Limitations
- Theory restricted to overparameterized linear regression with top-1 routing; top-k routing is not analyzed.
- Real-data experiments are on small-scale benchmarks with ResNet-18 and a small CNN and use a simplified task setup.
Experimental setup
Datasets: CIFAR-10 MNIST CIFAR-100 Tiny ImageNet synthetic linear regression data
Benchmarks: CIFAR-10 MNIST CIFAR-100 Tiny ImageNet synthetic linear regression
Metrics: forgetting overall generalization error average incremental accuracy
Baselines: single expert (M=1)
Builds on: mixture-of-experts with switch routing minimum-norm overparameterized linear regression
Closest papers by meaning
- Training Consistent Mixture-of-Experts-Based Prompt Generator for Continual Learning (AAAI 2025 · similarity 0.88)
- Mixture of Experts Meets Prompt-Based Continual Learning (NeurIPS 2024 · similarity 0.88)
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient (ICML 2025 · similarity 0.88)
- Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation (CVPR 2024 · similarity 0.88)
- MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts (AAAI 2024 · similarity 0.88)
- Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspective (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.