Research map / Label-efficient and robust learning / Knowledge distillation
Efficient Knowledge Distillation from Model Checkpoints
Chaofei Wang, Qisen Yang, Rui Huang, Shiji Song, Gao Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use half-way intermediate teacher instead of fully converged teacher | KD · CIFAR-100 WRN-40-2 -> MobileNetV2 | top-1 accuracy %: 68.03 -> 68.99 | Improved |
| use half-way intermediate teacher instead of fully converged teacher | KD · CIFAR-100 ResNet-110 -> ResNet-32 | top-1 accuracy %: 72.48 -> 72.49 | No clear effect |
| use half-way intermediate teacher instead of fully converged teacher | KD · ImageNet ResNet-34 -> ResNet-18 | top-1 accuracy %: 70.66 -> 70.86 | Improved |
| use best intermediate checkpoint instead of final teacher | KD · CIFAR-100 WRN-40-2 -> WRN-40-1 | top-1 accuracy %: 72.68 -> 73.26 | Improved |
| use Snapshot Ensemble (full + own intermediate) instead of Full Ensemble teacher | Ensemble KD · Average over CIFAR-100 and Tiny-ImageNet | top-1 accuracy %: 66.11 -> 66.63 | Improved |
| use Snapshot Ensemble instead of Full Ensemble as teacher | teacher ensemble · Average ensemble accuracy | top-1 accuracy %: 73.06 -> 69.95 | Hurt |
| select teacher checkpoint by maximizing I(X;F)+I(Y;F) (ablation) | KD · CIFAR-100 average of four pairs | top-1 accuracy %: 70.46 -> 70.87 | Improved |
| use half-way checkpoint T0.5 instead of full teacher (ablation) | KD · CIFAR-100 average of four pairs | top-1 accuracy %: 70.46 -> 70.76 | Improved |
| use early checkpoint T0.3 (60th epoch) instead of full teacher (ablation) | KD · CIFAR-100 average of four pairs | top-1 accuracy %: 70.46 -> 69.78 | Hurt |
| increase Snapshot Ensemble size k at same size as Full Ensemble (ablation) | Ensemble KD · CIFAR-100 WRN-40-2 -> WRN-40-1 | top-1 accuracy % | Improved |
Closest papers by meaning
- Learning Student-Friendly Teacher Networks for Knowledge Distillation (NeurIPS 2021 · similarity 0.90)
- Does Knowledge Distillation Really Work? (NeurIPS 2021 · similarity 0.90)
- Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation (NeurIPS 2022 · similarity 0.90)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.89)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.88)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
