Research map / Label-efficient and robust learning / Knowledge distillation
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, Kaisheng Ma
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add self distillation (auxiliary classifiers trained from deepest one) | ResNet50 · CIFAR100 | top-1 accuracy (%): 77.68 -> 80.56 | Improved |
| add self distillation | ResNet50 · ImageNet val | top-1 accuracy (%): 73.56 -> 75.24 | Improved |
| add self distillation | ResNet18 · ImageNet val | top-1 accuracy (%): 68.12 -> 69.84 | Improved |
| add self distillation | ResNeXt29-8 · CIFAR100 | top-1 accuracy (%): 81.29 -> 81.51 | Improved |
| use self distillation instead of deep supervision (DSN) (ablation) | ResNet50 · CIFAR100 | top-1 accuracy (%): 80.27 -> 80.56 | Improved |
| use self distillation instead of deep supervision (DSN), shallowest classifier (ablation) | ResNet152 · CIFAR100 | classifier 1/4 accuracy (%): 67.60 -> 68.84 | Improved |
| use self distillation instead of traditional KD with teacher ResNet152 (ablation) | ResNet50 student · CIFAR100 | top-1 accuracy (%): 79.33 -> 80.56 | Improved |
| use self distillation instead of AT with a WideResNet44-8 teacher (ablation) | WideResNet20-8 · CIFAR100 | top-1 accuracy (%): 80.65 -> 80.92 | Improved |
| replace pre-trained teacher training with one-step self distillation (ablation) | ResNet50 · CIFAR100 | training time: 26.98 hours -> 5.87 hours | Improved |
| add ensemble of all classifiers' softmax outputs (ablation) | self distillation ResNet50 · ImageNet val | top-1 accuracy (%): 75.24 -> 74.73 | Hurt |
| add ensemble of all classifiers' softmax outputs (ablation) | self distillation ResNet50 · CIFAR100 | top-1 accuracy (%): 80.56 -> 81.04 | Improved |
| exit at shallow classifier 1/4 instead of the full network (ablation) | self distillation ResNet50 · CIFAR100 | top-1 accuracy (%): 77.68 -> 68.23 | Hurt |
Closest papers by meaning
- Self-Distillation: Towards Efficient and Compact Neural Networks (TPAMI 2021 · similarity 0.95)
- Data-Distortion Guided Self-Distillation for Deep Neural Networks (AAAI 2019 · similarity 0.92)
- Knowledge Distillation: A Survey (IJCV 2021 · similarity 0.91)
- Improved Knowledge Distillation via Teacher Assistant (AAAI 2020 · similarity 0.90)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.90)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
