Research map / Architectures and efficiency / RNNs and normalization
Harmonized Dense Knowledge Distillation Training for Multi-Exit Architectures
Xinglu Wang, Yingming Li
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dense distillation with harmonized adaptive loss weights | MSDNet · ImageNet(full), Exit-1 | top-5 accuracy: 80.15 -> 82.44 | Improved |
| Use dense distillation with harmonized adaptive loss weights | MSDNet · ImageNet(full), Exit-5 | top-5 accuracy: 91.74 -> 92.69 | Improved |
| Use dense distillation with harmonized adaptive loss weights | MSDNet · CIFAR(500), Exit-1 | top-5 accuracy: 86.47 -> 87.64 | Improved |
| Use dense distillation with harmonized adaptive loss weights | MSDNet · CIFAR(500), Exit-8 | top-5 accuracy: 93.46 -> 94.02 | Improved |
| Change the initial loss-weight configuration (ablation) | HDKD · ImageNet(300), Exit-1 | top-5 accuracy: 78.72 -> 79.22 | No clear effect |
| Keep the final learned loss weights fixed from the start instead of evolving them during training (ablation) | HDKD · ImageNet(300), Exit-5 | top-5 accuracy: 87.37 -> 83.33; 87.37 -> 87.94 | Hurt |
| Remove the ratio regularizer from the validation loss (ablation) | HDKD · ImageNet(300), Exit-5 | top-5 accuracy: 87.37 -> 86.11 | Hurt |
| Use only validation cross-entropy, without the ratio regularizer (ablation) | HDKD · ImageNet(300), Exit-1 | top-5 accuracy: 78.72 -> 79.55 | Improved |
Closest papers by meaning
- Distillation-Based Training for Multi-Exit Architectures (ICCV 2019 · similarity 0.89)
- SDPGO: Efficient Self-Distillation Training Meets Proximal Gradient Optimization (NeurIPS 2025 · similarity 0.84)
- NEO-KD: Knowledge-Distillation-Based Adversarial Training for Robust Multi-Exit Neural Networks (NeurIPS 2023 · similarity 0.84)
- Harmonizing knowledge Transfer in Neural Network with Unified Distillation (ECCV 2024 · similarity 0.84)
- Knowledge Transfer via Dense Cross-Layer Mutual-Distillation (ECCV 2020 · similarity 0.84)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
