Research map / Label-efficient and robust learning / Knowledge distillation
Densely Guided Knowledge Distillation Using Multiple Teacher Assistants
Wonchul Son, Jaemin Na, Junyong Choi, Wonjun Hwang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| guide each TA and the student from all higher-level TAs and the teacher (dense guidance) instead of only the previous TA (ablation) | TAKD, plain CNN full path · CIFAR-100 | accuracy: 45.14 -> 48.92 | Improved |
| dense guidance instead of sequential guidance, ResNet path with two TAs (ablation) | TAKD, ResNet · CIFAR-10 | accuracy: 88.01 -> 89.02 | Improved |
| dense guidance instead of sequential guidance on ImageNet with a single TA (ablation) | TAKD, ResNet-18 student · ImageNet val | top-1 accuracy: 71.37 -> 71.73 | Improved |
| lengthen the TAKD distillation path from 3 TAs to 7 TAs (ablation) | TAKD, plain CNN · CIFAR-100 | accuracy: 45.14 -> 44.07 | Hurt |
| lengthen the DGKD distillation path from 3 TAs to 7 TAs (ablation) | DGKD, plain CNN · CIFAR-100 | accuracy: 48.92 -> 49.56 | Improved |
| add stochastic dropping of teacher/TA knowledge (p=0.75) (ablation) | DGKD, plain CNN · CIFAR-100 | accuracy: 48.92 -> 50.15 | Improved |
| add stochastic dropping of teacher/TA knowledge (p=0.75) (ablation) | DGKD, ResNet · CIFAR-10 | accuracy: 89.02 -> 89.66 | Improved |
| lower survival probability to 0.25 (dropping three of four teachers) (ablation) | Stochastic DGKD · CIFAR-100 | accuracy | Hurt |
| use survival probability between 0.75 and 0.5 instead of 1 (ablation) | DGKD · CIFAR-100 | accuracy | Improved |
| use an ensemble of four independently trained 10-layer teachers instead of KD from one teacher (ablation) | KD, plain CNN student · CIFAR-100 | accuracy: 42.56 -> 42.57 | No clear effect |
| use DGKD instead of an ensemble of T10, T8, T6, T4 (ablation) | plain CNN student · CIFAR-100 | accuracy: 43.25 -> 48.92 | Improved |
| use two TAs instead of one TA in the TAKD path (ablation) | TAKD, plain CNN · CIFAR-100 | accuracy | Hurt |
Closest papers by meaning
- Improved Knowledge Distillation via Teacher Assistant (AAAI 2020 · similarity 0.95)
- Knowledge Distillation: A Survey (IJCV 2021 · similarity 0.93)
- Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation (NeurIPS 2022 · similarity 0.92)
- Learning Student-Friendly Teacher Networks for Knowledge Distillation (NeurIPS 2021 · similarity 0.92)
- Knowledge Transfer via Dense Cross-Layer Mutual-Distillation (ECCV 2020 · similarity 0.92)
- Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher (NeurIPS 2020 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
