Research map / Label-efficient and robust learning / Knowledge distillation
Does Knowledge Distillation Really Work?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase teacher ensemble size from 1 to 5 components | Knowledge distillation into a single student · IMDB test | teacher-student agreement: 86.488 (0.521) -> 90.793 (0.180) | Helped |
| Increase teacher ensemble size from 1 to 5 components | Knowledge distillation into a single student · ImageNet test | teacher-student agreement: 0.855 (0.001) -> 0.884 (0.001) | Helped |
| Increase teacher ensemble size from 1 to 5 components | Knowledge distillation into a single student · IMDB test | accuracy: 80.353 (0.198) -> 81.167 (0.196) | Mixed |
| Increase teacher ensemble size from 1 to 5 components | Knowledge distillation into a single student · ImageNet test | accuracy: 0.753 (0.001) -> 0.756 (0.001) | Helped |
| Distill on teacher-training data rather than reserved data (ablation) | CIFAR-100 ResNet-56 distillation · CIFAR-100 test | accuracy | Helped |
| Distill on teacher-training data rather than reserved data (ablation) | CIFAR-100 ResNet-56 distillation · CIFAR-100 test | teacher-student agreement | Hurt |
| Add both teacher-training and reserved data to distillation (ablation) | CIFAR-100 ResNet-56 distillation · CIFAR-100 test | teacher-student agreement | Helped |
| Use MixUp with temperature 4 instead of baseline augmentation (ablation) | ResNet-56 student distilling a 5-component ResNet-56 ensemble · CIFAR-100 test | teacher-student agreement: 86% | Helped |
| Use noise or out-of-distribution augmentation (ablation) | CIFAR-100 ResNet-56 distillation · CIFAR-100 | performance | Hurt |
| Train for 5k rather than 300 epochs (ablation) | ResNet-20 self-distillation · CIFAR-100 train | teacher-student agreement: 78.95% -> 83.3% | Helped |
| Switch from SGD to Adam (ablation) | ResNet-20 self-distillation · CIFAR-100 | teacher-student agreement | Hurt |
| Initialize student with teacher's initial weights (ablation) | ResNet-20 self-distillation · CIFAR-100 test | teacher-student agreement: 77.174 (0.352) -> 77.098 (0.238) | No clear effect |
Closest papers by meaning
- Learning Student-Friendly Teacher Networks for Knowledge Distillation (NeurIPS 2021 · similarity 0.93)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.93)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.90)
- Knowledge Distillation Based on Transformed Teacher Matching (ICLR 2024 · similarity 0.90)
- Strong Teacher Not Needed? On Distillation in LLM Pretraining (NeurIPS 2026 · similarity 0.89)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.