Research map / Continual learning / Online and replay-based
Divide and not forget: Ensemble of selectively trained experts in Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEED, an exemplar-free class-incremental learning ensemble where each task fine-tunes only one selected expert (chosen by minimum KL overlap of per-class Gaussians) and inference uses Bayes classification over all experts.
Problem
Exemplar-free class-incremental learning with equal-size tasks is hard because feature extractors trained on small early tasks lack plasticity, while full-ensemble methods still suffer forgetting and computational cost.
Approach
Maintain K experts sharing frozen initial layers, each with per-class multivariate Gaussians in its latent space; for each new task, select the single expert where new-class Gaussians overlap least (symmetrized KL) and fine-tune only it with CE plus feature distillation; predict by averaging softmaxed Gaussian log-likelihoods across experts.
Key results
- On CIFAR-100 equal split with T=10, SEED reaches 61.7% task-agnostic average incremental accuracy versus 47.0% for the best baseline LwF* (FeTrIL 46.3%).
- On DomainNet T=12, SEED reaches 45.0% vs 33.5% for FeTrIL.
- In the large-first-task CIFAR-100 setting, SEED is 4.6 points above FeTrIL at T=6 but FeTrIL is better at T=11 and T=21.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SEED instead of FeTrIL on equal-split CIFAR-100, DomainNet, and ImageNet-Subset | exemplar-free class-incremental learning · CIFAR-100 T=10/20/50; DomainNet T=12/24/36; ImageNet-Subset T=10 | average incremental accuracy (%): 61.7$\pm$0.4 vs 46.3$\pm$0.3; 56.2$\pm$0.3 vs 38.7$\pm$0.3; 42.6$\pm$1.4 vs 27.0$\pm$1.2; 45.0$\pm$0.2 vs 33.5$\pm$0.6; | Helped |
| Use SEED instead of FeTrIL on CIFAR-100 with a larger first task | SEED · CIFAR-100 large first task, T=6/11/21 | average incremental accuracy (%): 70.9$\pm$0.3 vs 66.3; 69.3$\pm$0.5 vs 65.2; 62.9$\pm$0.9 vs 61.5 | Helped |
| Use SEED rather than FeTrIL with a large first task | SEED · ImageNet-Subset large first task, T=6/11/21 | average incremental accuracy (%): 75.5$\pm$0.4 vs 72.2; 70.9$\pm$0.5 vs 71.2; 63.0$\pm$0.8 vs 67.1 | Mixed |
| Share more layers across experts (ablation) | SEED · CIFAR-100 task-aware, 20-split and 50-split | average task-aware accuracy (%): 86.8$\pm$0.3 -> 85.6$\pm$0.3; 91.2$\pm$0.4 -> 89.6$\pm$0.2 | Hurt |
| Replace SEED selective training with a standard ensemble (ablation) | SEED · CIFAR-100 CIL, T=10 | average incremental accuracy (%): 61.7 $\pm$0.4 -> 56.9$\pm$0.4 | Hurt |
| Weight ensemble predictions (ablation) | SEED · CIFAR-100 CIL, T=10 | average incremental accuracy (%): 61.7 $\pm$0.4 -> 57.0$\pm$0.5 | Hurt |
| Use CoSCL ensemble cooperation (ablation) | SEED · CIFAR-100 CIL, T=10 | average incremental accuracy (%): 61.7 $\pm$0.4 -> 57.3$\pm$0.4 | Hurt |
| Replace multivariate Gaussian class distributions with diagonal Gaussians (ablation) | SEED · CIFAR-100 CIL, T=10 | average incremental accuracy (%): 61.7 $\pm$0.4 -> 53.5$\pm$0.5 | Hurt |
| Remove covariance and represent classes with mean prototypes (ablation) | SEED · CIFAR-100 CIL, T=10 | average incremental accuracy (%): 61.7 $\pm$0.4 -> 54.1$\pm$0.3 | Hurt |
| Remove temperature from softmax (ablation) | SEED · CIFAR-100 CIL, T=10 | average incremental accuracy (%): 61.7 $\pm$0.4 -> 59.2$\pm$0.5 | Hurt |
| Add ReLU activation at the last layer (ablation) | SEED · CIFAR-100 CIL, T=10 | average incremental accuracy (%): 61.7 $\pm$0.4 -> 57.8$\pm$0.6 | Hurt |
| Represent each class with a mixture of 2 multivariate Gaussians (ablation) | SEED · CIFAR-100 CIL, T=10 | average incremental accuracy (%): 61.7 ±0.5 -> 60.5±0.7 | Hurt |
Surprising findings
- Naive ensemble averaging of all experts reduced accuracy by about 4.8 points, showing the selective training rather than ensembling alone drives gains.
- Selecting the expert with maximum overlap (KL-min) was not used; the minimum-overlap selection outperformed random, round-robin, and maximum-overlap strategies.
Limitations
- Requires the maximum number of experts to be specified upfront.
- Not suitable when tasks are unrelated and parameter budgets are tight, since shared initial layers may hurt performance.
- Class covariance can be singular, requiring reduced latent dimensionality.
Experimental setup
Datasets: CIFAR-100 ImageNet-Subset DomainNet
Benchmarks: CIFAR-100 ImageNet-Subset DomainNet
Metrics: average incremental accuracy forgetting intransigence
Baselines: EWC LwF PASS IL2A SSRE FeTrIL CoSCL HAT MARK BNS SDC ABD
Builds on: CoSCL FeTrIL LwF EWC FACIL
Closest papers by meaning
- Training Consistent Mixture-of-Experts-Based Prompt Generator for Continual Learning (AAAI 2025 · similarity 0.88)
- Theory on Mixture-of-Experts in Continual Learning (ICLR 2025 · similarity 0.88)
- Mixture of Experts Meets Prompt-Based Continual Learning (NeurIPS 2024 · similarity 0.86)
- Consistent Prompting for Rehearsal-Free Continual Learning (CVPR 2024 · similarity 0.86)
- Continual learning with the neural tangent ensemble (NeurIPS 2024 · similarity 0.85)
- MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.