Research map / Label-efficient and robust learning / Knowledge distillation
On the Efficacy of Knowledge Distillation
Jang Hyun Cho, Bharath Hariharan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| stop student distillation early (ESKD) instead of full KD, ResNet18 teacher (ablation) | ResNet18 student · ImageNet val | top-1 error (%): 30.57 -> 29.01 | Improved |
| stop student distillation early (ESKD) instead of full KD, ResNet34 teacher (ablation) | ResNet18 student · ImageNet val | top-1 error (%): 30.79 -> 29.16 | Improved |
| stop student distillation early (ESKD) instead of full KD, ResNet50 teacher (ablation) | ResNet18 student · ImageNet val | top-1 error (%): 30.95 -> 29.35 | Improved |
| use a larger, more accurate teacher in full KD (ResNet18 to ResNet50) | ResNet18 student · ImageNet val | top-1 error (%): 30.57 -> 30.95 | Hurt |
| early-stop the teacher training (WRN16-8, 20/65) (ablation) | WRN16-1 student KD · CIFAR10 | test error (%): 8.028 -> 7.482 | Improved |
| early-stop the teacher training (WRN16-3, 20/65) (ablation) | WRN16-1 student KD · CIFAR10 | test error (%): 7.681 -> 7.498 | Improved |
| early-stop the teacher training (ResNet152, 35 epochs) with ESKD (ablation) | ResNet18 student · ImageNet val | top-1 error (%): 29.45 -> 29.26 | Improved |
| add attention transfer to ESKD (ResNet34 teacher) | ResNet18 student · ImageNet val | top-1 error (%): 29.16 -> 28.84 | Improved |
| add early-stopped teacher (50 epochs) to ESKD + AT (ablation) | ResNet18 student · ImageNet val | top-1 error (%): 28.84 -> 28.61 | Improved |
| add an intermediate medium teacher (Large->Med->Small) to distill to the small student (ablation) | WRN16-1 student · CIFAR10 | test error (%): 7.614 -> 8.04 | Hurt |
| train the teacher with 5 sequential KD iterations instead of from scratch (ablation) | WRN16-3 teacher for WRN16-1 · CIFAR10 | student test error (%): 7.61 -> 7.79 | No clear effect |
| ensemble sequential KD generations instead of ensemble of scratch models | ResNet8 · CIFAR10 | ensemble error (%): 10.176 -> 12.786 | Hurt |
Closest papers by meaning
- Strong Teacher Not Needed? On Distillation in LLM Pretraining (NeurIPS 2026 · similarity 0.90)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.89)
- Revisit the Essence of Distilling Knowledge through Calibration (ICML 2024 · similarity 0.89)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.89)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.88)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
