Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Transfer via Dense Cross-Layer Mutual-Distillation
Anbang Yao, Dawei Sun
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add deep-supervision auxiliary classifiers and dense cross-layer KD to DML (DCM) | DML · CIFAR-100, two WRN-28-10 | top-1 error: 17.89 -> 16.61 | Improved |
| add auxiliary classifiers with only the last-layer KD (DML + DS) (ablation) | DML · CIFAR-100, two DenseNet-40-12 | top-1 error: 23.18 -> 23.08 | No clear effect |
| add same-staged cross-layer KD (DCM-1) to DML+DS (ablation) | DML + DS · CIFAR-100, two DenseNet-40-12 | top-1 error: 23.18 -> 22.86 | Improved |
| add different-staged cross-layer KD (DCM-2) instead of same-staged (ablation) | DCM · CIFAR-100, two DenseNet-40-12 | top-1 error: 22.86 -> 22.43 | Improved |
| use both same-staged and different-staged KD (full DCM) rather than DCM-2 only (ablation) | DCM · CIFAR-100, two DenseNet-40-12 | top-1 error: 22.43 -> 22.35 | Improved |
| add independent deep supervision only (no mutual distillation) | ResNet-164 · CIFAR-100 | top-1 error: 22.56 -> 21.38 | Improved |
| add DCM versus DS alone (ablation) | WRN-28-10 · CIFAR-100 | top-1 error: 18.32 -> 16.61 | Improved |
| use 2 auxiliary classifiers instead of 1 (ablation) | DCM · CIFAR-100, two WRN-28-10 | top-1 error: 17.40 -> 16.61 | Improved |
| use 3 auxiliary classifiers instead of 2 (ablation) | DCM · CIFAR-100, two WRN-28-10 | top-1 error: 16.61 -> 16.59 | No clear effect |
| use default strong auxiliary classifiers instead of simple APFC ones (ablation) | DCM · CIFAR-100, DenseNet-40-12 & WRN-28-10 | top-1 error: 25.10 -> 22.35 | Improved |
| use default width auxiliary classifiers instead of narrow ones (ablation) | DCM · CIFAR-100, DenseNet-40-12 & WRN-28-10 | top-1 error: 16.88 -> 16.61 | Improved |
| train ResNet-110 with KD from pre-trained WRN-28-10 teacher | ResNet-110 · CIFAR-100 | top-1 error: 26.55 -> 26.66 | No clear effect |
Closest papers by meaning
- Densely Guided Knowledge Distillation Using Multiple Teacher Assistants (ICCV 2021 · similarity 0.92)
- Revisiting Knowledge Distillation: An Inheritance and Exploration Framework (CVPR 2021 · similarity 0.92)
- Deep Mutual Learning (CVPR 2018 · similarity 0.91)
- Harmonizing knowledge Transfer in Neural Network with Unified Distillation (ECCV 2024 · similarity 0.91)
- A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning (CVPR 2017 · similarity 0.91)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
