Research map / Architectures and efficiency / RNNs and normalization
Distillation-Based Training for Multi-Exit Architectures
Mary Phuong, Christoph H. Lampert
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with distillation from the last exit instead of exit-wise loss only | MSDNet · ImageNet(100) Exit 1 | top-5 accuracy (%): 64.4 -> 68.1 | Improved |
| train with distillation from the last exit instead of exit-wise loss only | MSDNet · ImageNet subsets | top-5 accuracy (%): accuracy gains of up to 3.8% | Improved |
| train with distillation from the last exit instead of exit-wise loss only | MSDNet · CIFAR100 subsets | top-5 accuracy (%): up to 6.3% improvement | Improved |
| train with distillation instead of exit-wise loss on full ImageNet | MSDNet · ImageNet(full) | top-5 accuracy (%): Exit 1: 87.8 -> 88.8; Exit 5: 92.3 -> 91.7 | Mixed |
| add unlabeled data to distillation (semi-supervised) (ablation) | MSDNet · CIFAR100 Exit 1 | top-5 accuracy (%): 1.2%, 0.9%, and 0.4% increase for CIFAR(150), CIFAR(250), CIFAR(350) | Improved |
| add unlabeled data to distillation (semi-supervised) (ablation) | MSDNet · ImageNet | top-5 accuracy (%): small additional improvement, e.g. ImageNet(100) Exit 5: 69.3 -> 69.7 | Improved |
| add unlabeled data to distillation (semi-supervised), anytime-mode (ablation) | MSDNet · CIFAR(150) late exits | top-5 accuracy (%) | Mixed |
| reduce amount of labeled training data (effect on distillation benefit) | MSDNet · ImageNet | top-5 accuracy (%): Exit 1: 64.4 -> 68.1 at ImageNet(100) vs 87.8 -> 88.8 at full | Improved |
| add adaptive temperature annealing instead of fixed temperature (ablation) | distillation-trained MSDNet · CIFAR(250) | top-5 accuracy (%) | No clear effect |
| choose different teacher exits instead of the last exit (ablation) | distillation-trained MSDNet · CIFAR100/ImageNet | top-5 accuracy (%) | No clear effect |
Closest papers by meaning
- Harmonized Dense Knowledge Distillation Training for Multi-Exit Architectures (AAAI 2021 · similarity 0.89)
- One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation (NeurIPS 2023 · similarity 0.87)
- DOT: A Distillation-Oriented Trainer (ICCV 2023 · similarity 0.87)
- How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies (ICML 2025 · similarity 0.87)
- Understanding the Role of the Projector in Knowledge Distillation (AAAI 2024 · similarity 0.86)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
