Research map / Label-efficient and robust learning / Knowledge distillation
Continual Distillation of Teachers from Different Domains
Nicolas Michel, Maorong Wang, Jiangpeng He, Toshihiko Yamasaki
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add related external data (CIFAR20 domain 4) to distillation (ablation) | KL-divergence distillation · CIFAR20 | avg accuracy domains 0-3: 61.94 -> 71.36 | Improved |
| use CUB images as external data (ablation) | KL-divergence distillation · CIFAR20 | avg accuracy domains 0-3: 61.94 -> 67.02 | Improved |
| use MNIST images as external data (unrelated domain) (ablation) | KL-divergence distillation · CIFAR20 | avg accuracy domains 0-3: 61.94 -> 59.78 | Hurt |
| increase external data ratio in distillation set (0% to 50% to 66%) (ablation) | KL-divergence student, teacher T01 · CIFAR20 domain 1 accuracy | accuracy on unseen domain 1: 37.80 -> 77.15 -> 85.20 | Improved |
| add self-distillation from previous checkpoint on external data only (SE2D vs Self-Distillation) | Self-Distillation · CIFAR20 + related ED | avg accuracy domains 0-3: 74.93 -> 76.17 | Improved |
| add SE2D vs Self-Distillation on Digits | Self-Distillation · Digits + KMNIST | avg accuracy: 85.58 -> 87.00 | Improved |
| add SE2D vs Self-Distillation on DomainNet | Self-Distillation · DomainNet + Sketch ED | avg accuracy: 48.76 -> 48.01 | Hurt |
| add SE2D vs Self-Distillation with unrelated MNIST ED | Self-Distillation · CIFAR20 + MNIST | avg accuracy domains 0-3: 54.26 -> 57.25 | Improved |
| add related external data to KL distillation on Digits (ablation) | KL-divergence distillation · Digits | avg accuracy: 73.40 -> 71.75 | Hurt |
| add related external data to DKD on Digits (ablation) | DKD · Digits MNIST-M | accuracy: 54.50 -> 33.84 | Hurt |
| add SE2D forgetting reduction on CIFAR20 vs KL | KL-divergence distillation · CIFAR20 + related ED | avg forgetting: 17.23 -> 4.44 | Improved |
| add SE2D forgetting reduction on Digits vs KL | KL-divergence distillation · Digits + KMNIST | avg forgetting: 19.17 -> 3.73 | Improved |
Closest papers by meaning
- Improved Knowledge Distillation via Teacher Assistant (AAAI 2020 · similarity 0.89)
- Updating Parametric Knowledge with Context Distillation Retains Post-Training Capabilities (ICML 2026 · similarity 0.89)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.89)
- Densely Guided Knowledge Distillation Using Multiple Teacher Assistants (ICCV 2021 · similarity 0.89)
- Similarity-Preserving Knowledge Distillation (ICCV 2019 · similarity 0.89)
- Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
