AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Densely Guided Knowledge Distillation Using Multiple Teacher Assistants

Wonchul Son, Jaemin Na, Junyong Choi, Wonjun Hwang

ICCV 2021 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 161 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
guide each TA and the student from all higher-level TAs and the teacher (dense guidance) instead of only the previous TA (ablation)TAKD, plain CNN full path · CIFAR-100accuracy: 45.14 -> 48.92Improved
dense guidance instead of sequential guidance, ResNet path with two TAs (ablation)TAKD, ResNet · CIFAR-10accuracy: 88.01 -> 89.02Improved
dense guidance instead of sequential guidance on ImageNet with a single TA (ablation)TAKD, ResNet-18 student · ImageNet valtop-1 accuracy: 71.37 -> 71.73Improved
lengthen the TAKD distillation path from 3 TAs to 7 TAs (ablation)TAKD, plain CNN · CIFAR-100accuracy: 45.14 -> 44.07Hurt
lengthen the DGKD distillation path from 3 TAs to 7 TAs (ablation)DGKD, plain CNN · CIFAR-100accuracy: 48.92 -> 49.56Improved
add stochastic dropping of teacher/TA knowledge (p=0.75) (ablation)DGKD, plain CNN · CIFAR-100accuracy: 48.92 -> 50.15Improved
add stochastic dropping of teacher/TA knowledge (p=0.75) (ablation)DGKD, ResNet · CIFAR-10accuracy: 89.02 -> 89.66Improved
lower survival probability to 0.25 (dropping three of four teachers) (ablation)Stochastic DGKD · CIFAR-100accuracyHurt
use survival probability between 0.75 and 0.5 instead of 1 (ablation)DGKD · CIFAR-100accuracyImproved
use an ensemble of four independently trained 10-layer teachers instead of KD from one teacher (ablation)KD, plain CNN student · CIFAR-100accuracy: 42.56 -> 42.57No clear effect
use DGKD instead of an ensemble of T10, T8, T6, T4 (ablation)plain CNN student · CIFAR-100accuracy: 43.25 -> 48.92Improved
use two TAs instead of one TA in the TAKD path (ablation)TAKD, plain CNN · CIFAR-100accuracyHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy