Research map / Continual learning / Class-incremental
Class Incremental Learning with Multi-Teacher Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MTD, which finds multiple diverse teachers from a single basic model via weight permutation, feature perturbation and diversity regularization, each teacher as a small branch, and uses them to improve distillation-based class incremental learning.
Problem
Class incremental learning suffers catastrophic forgetting; existing distillation methods inherit knowledge from a single teacher, and retraining multiple diverse teachers is time- and memory-consuming.
Approach
Generate diverse teachers cheaply by weight-permuting a basic model's branch (teleporting parameters to another low-loss region), perturbing its input features to break output equivalence, and minimizing absolute cosine similarity between teacher embeddings; teachers are lightweight branches sharing the feature extractor.
Key results
- MTD-T improves PODNet by 2.57%, 2.65%, and 1.49% on CIFAR-100 B50 for 5, 10, and 25 steps.
- MTD-T improves PODNet by 5.49% on ImageNet-100 B50 with 25 steps.
- MTD-T improves AFC by 1.15%, 0.74%, and 0.68% on CIFAR-100 B0 for 5, 10, and 20 tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MTD and use the teacher ensemble instead of the student | PODNet · CIFAR-100 B50-5S | average incremental accuracy: 65.07(˘0.28) -> 67.64(˘0.21) | Helped |
| Apply MTD and use the teacher ensemble instead of the student | PODNet · CIFAR-100 B50-10S | average incremental accuracy: 62.93(˘0.14) -> 65.58(˘0.20) | Helped |
| Apply MTD and use the teacher ensemble instead of the student | PODNet · CIFAR-100 B50-25S | average incremental accuracy: 59.45(˘0.28) -> 60.94(˘0.22) | Helped |
| Add feature perturbation (ablation) | PODNet · CIFAR-100 B50-5S | average incremental accuracy: 65.07(˘0.28) -> 66.76(˘0.14) | Helped |
| Add weight permutation after feature perturbation (ablation) | PODNet · CIFAR-100 B50-5S | average incremental accuracy: 66.76(˘0.14) -> 66.99(˘0.23) | Helped |
| Add diversity regularization to feature perturbation (ablation) | PODNet · CIFAR-100 B50-5S | average incremental accuracy: 66.76(˘0.14) -> 67.53(˘0.08) | Helped |
| Use more than two teachers (ablation) | MTD-S · CIFAR-100 B50 | average incremental accuracy | Hurt |
Surprising findings
- Weight permutation alone brings only small gains, attributed to random permutation not matching the separation pattern of independently trained teachers.
- Adding more than two teachers degrades performance, since averaging logits becomes crude and teacher diversity decreases.
Limitations
- Permutation matrices are chosen randomly and may not match the true separation pattern of independently trained teachers.
- Averaging logits of more than two teachers is crude and diversity drops as teacher count grows.
Experimental setup
Datasets: CIFAR-100 ImageNet-100 ImageNet-1000
Benchmarks: CIFAR-100 ImageNet-100 ImageNet-1000
Metrics: average incremental accuracy
Baselines: PODNet AFC BiC Mnemonics GeoDL iCaRL LUCIR AANet DT-CIL
Builds on: PODNet AFC LwF iCaRL LUCIR
Closest papers by meaning
- Defying Imbalanced Forgetting in Class Incremental Learning (AAAI 2024 · similarity 0.88)
- Make Domain Shift a Catastrophic Forgetting Alleviator in Class-Incremental Learning (AAAI 2025 · similarity 0.88)
- Gradient Reweighting: Towards Imbalanced Class-Incremental Learning (CVPR 2024 · similarity 0.88)
- Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition (AAAI 2025 · similarity 0.87)
- Long-Tail Class Incremental Learning via Independent Sub-prototype Construction (CVPR 2024 · similarity 0.87)
- Maintaining Fairness in Logit-based Knowledge Distillation for Class-Incremental Learning (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.