Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Distillation: Bad Models Can Be Good Role Models
Gal Kaplun, eran malach, Preetum Nakkiran, Shai Shalev-Shwartz
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace one teacher with an ensemble of 10 teachers at inference | ResNet-18 teacher · CIFAR-10 test | accuracy: 81.3% -> 87.8% | Improved |
| Distill using ensemble-labeled unlabeled examples | ResNet-18 teacher ensemble · CIFAR-10 test | accuracy: 81.3% -> about 90% | Improved |
| Distill using examples labeled by a randomly selected teacher | ResNet-18 teacher pool · CIFAR-10 test | accuracy: 81.3% -> about 90% | Improved |
| Use ensemble labeling instead of randomly selected teacher labeling | CIFAR-5m student distillation · CIFAR-10 test | accuracy | No clear effect |
| Increase the number of teachers | Ensemble pseudo-labeling and random pseudo-labeling · CIFAR-10 test | accuracy | Improved |
Closest papers by meaning
- Towards Understanding Knowledge Distillation (ICML 2019 · similarity 0.89)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.87)
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.86)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.86)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.86)
- High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
