Research map / Architectures and efficiency / RNNs and normalization
Revisit the Power of Vanilla Knowledge Distillation: from Small Scale to Large Scale
Zhiwei Hao, Jianyuan Guo, Kai Han, Han Hu, Chang Xu, Yunhe Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| stronger recipe (AdamW, 300 epochs) vs previous recipe (ablation) | vanilla KD · Res50-MBNetV2 ImageNet-1K | top-1 accuracy (%): 68.58 -> 74.23 | Improved |
| stronger recipe (AdamW, 300 epochs) vs previous recipe (ablation) | vanilla KD · Res34-Res18 ImageNet-1K | top-1 accuracy (%): 70.66 -> 73.46 | Improved |
| stronger recipe with 2400 epochs on CIFAR-100 (ablation) | vanilla KD · Res56-Res20 CIFAR-100 | gap to best method: 1.31% to 2.17% | Hurt |
| replace hint-based distillation by logits-based | vanilla KD · ImageNet-1K Res152 teacher A2 | top-1 accuracy (%): 79.55 -> 80.55 | Improved |
| replace multi-class CE by BCE for hard label (ablation) | vanilla KD · ImageNet-1K 300ep | top-1 accuracy (%): 80.49 -> 80.89 | Improved |
| replace KL by binary KL (BKL) for soft label (ablation) | vanilla KD · ImageNet-1K 300ep | top-1 accuracy (%): 80.89 -> 79.36 | Hurt |
| remove hard-label supervision (ablation) | vanilla KD · ImageNet-1K 600ep | top-1 accuracy (%): 81.68 -> 81.65 | No clear effect |
| vary learning rate and weight decay (ablation) | vanilla KD · ImageNet-1K Res50 A2 | top-1 accuracy (%): 79.77 -> 80.89 | No clear effect |
| increase training epochs from 300 to 4800 (ablation) | vanilla KD · Res50 student | top-1 accuracy (%): 80.96 -> 83.08 | Improved |
| increase epochs from 300 to 4800 (ablation) | vanilla KD · ViT-S student | top-1 accuracy (%): 81.38 -> 84.33 | Improved |
| use larger BEiTv2-L teacher instead of BEiTv2-B (ablation) | vanilla KD · Res50, 1200ep | top-1 accuracy (%): 82.35 -> 82.27 | Hurt |
| teacher pre-trained on IN-21K instead of IN-1K (ablation) | vanilla KD · Res50 student | top-1 accuracy (%): 81.97 -> 82.35 | Improved |
Closest papers by meaning
- Toward Effective Knowledge Distillation: Navigating Beyond Small-Data Pitfall (TPAMI 2025 · similarity 0.92)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.89)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.89)
- Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.88)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.88)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
