Research map / Label-efficient and robust learning / Knowledge distillation
What Knowledge Gets Distilled in Knowledge Distillation?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill from a color-invariant teacher rather than train the student independently | ResNet18 · ImageNet color transformations | color agreement: 71.3% -> 82.1% (KL); 71.3% -> 79.5% (CRD) | Helped |
| Distill from the teacher's own successful adversarial examples | CNN student, across CIFAR-100 teacher-student settings · CIFAR-100 | fooling rate: 31.23 -> 48.62 (KL), 62.63 (Hint), 49.41 (CRD); 42.42 -> 51.87, 49.79, 54.68; 36.57 -> 43.32, 43.91, 46.07 | Mixed |
| Distill teacher predictions or features into a student | ImageNet classification models · ImageNet CAM localization | teacher-student CAM similarity | Helped |
| Distill from a CNN teacher rather than train the CNN student independently | ResNet18 student · Teacher-crafted ImageNet adversarial images | fooling rate: 44.2%; 48%-52% | Helped |
| Distill from a transformer teacher into a CNN student | ResNet18 student · Teacher-crafted adversarial images | fooling rate | No clear effect |
| Distill from a teacher trained on additional domains while training only on MNIST-Orig | MNIST student · MNIST-Color and MNIST-M | top-1 accuracy | Helped |
| Distill from the teacher rather than train independently | FairFace ResNet20 student · FairFace race subgroups | accuracy across races | Mixed |
| Use deeper teacher features for Hint, rather than a middle-layer feature (ablation) | Hint distillation · ImageNet color invariance | color agreement | Helped |
| Replace teacher soft labels with noisy ground-truth-derived soft labels (ablation) | ResNet18 pseudo-KL student · ImageNet color invariance | color agreement: 71.10% vs 70.49% | Hurt |
| Distill crop invariance with KL or CRD rather than train independently | ImageNet ResNet18 student · ImageNet random crops | agreement score | Helped |
| Use Hint rather than KL or CRD for crop invariance transfer | ImageNet ResNet18 student · ImageNet random crops | agreement score | No clear effect |
| Distill from an anti-aliased shift-invariant teacher using KL or CRD | ResNet18 student · ImageNet shifts | agreement score | Helped |
Closest papers by meaning
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.91)
- Does Knowledge Distillation Really Work? (NeurIPS 2021 · similarity 0.91)
- Learning Student-Friendly Teacher Networks for Knowledge Distillation (NeurIPS 2021 · similarity 0.90)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.89)
- Revisit the Essence of Distilling Knowledge through Calibration (ICML 2024 · similarity 0.88)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.