AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Continual Distillation of Teachers from Different Domains

Nicolas Michel, Maorong Wang, Jiangpeng He, Toshihiko Yamasaki

CVPR 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add related external data (CIFAR20 domain 4) to distillation (ablation)KL-divergence distillation · CIFAR20avg accuracy domains 0-3: 61.94 -> 71.36Improved
use CUB images as external data (ablation)KL-divergence distillation · CIFAR20avg accuracy domains 0-3: 61.94 -> 67.02Improved
use MNIST images as external data (unrelated domain) (ablation)KL-divergence distillation · CIFAR20avg accuracy domains 0-3: 61.94 -> 59.78Hurt
increase external data ratio in distillation set (0% to 50% to 66%) (ablation)KL-divergence student, teacher T01 · CIFAR20 domain 1 accuracyaccuracy on unseen domain 1: 37.80 -> 77.15 -> 85.20Improved
add self-distillation from previous checkpoint on external data only (SE2D vs Self-Distillation)Self-Distillation · CIFAR20 + related EDavg accuracy domains 0-3: 74.93 -> 76.17Improved
add SE2D vs Self-Distillation on DigitsSelf-Distillation · Digits + KMNISTavg accuracy: 85.58 -> 87.00Improved
add SE2D vs Self-Distillation on DomainNetSelf-Distillation · DomainNet + Sketch EDavg accuracy: 48.76 -> 48.01Hurt
add SE2D vs Self-Distillation with unrelated MNIST EDSelf-Distillation · CIFAR20 + MNISTavg accuracy domains 0-3: 54.26 -> 57.25Improved
add related external data to KL distillation on Digits (ablation)KL-divergence distillation · Digitsavg accuracy: 73.40 -> 71.75Hurt
add related external data to DKD on Digits (ablation)DKD · Digits MNIST-Maccuracy: 54.50 -> 33.84Hurt
add SE2D forgetting reduction on CIFAR20 vs KLKL-divergence distillation · CIFAR20 + related EDavg forgetting: 17.23 -> 4.44Improved
add SE2D forgetting reduction on Digits vs KLKL-divergence distillation · Digits + KMNISTavg forgetting: 19.17 -> 3.73Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy