AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

On the Efficacy of Knowledge Distillation

Jang Hyun Cho, Bharath Hariharan

ICCV 2019 · Label-efficient and robust learning › Knowledge distillation · approach: Distillation with causal and adaptive guidance · 737 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
stop student distillation early (ESKD) instead of full KD, ResNet18 teacher (ablation)ResNet18 student · ImageNet valtop-1 error (%): 30.57 -> 29.01Improved
stop student distillation early (ESKD) instead of full KD, ResNet34 teacher (ablation)ResNet18 student · ImageNet valtop-1 error (%): 30.79 -> 29.16Improved
stop student distillation early (ESKD) instead of full KD, ResNet50 teacher (ablation)ResNet18 student · ImageNet valtop-1 error (%): 30.95 -> 29.35Improved
use a larger, more accurate teacher in full KD (ResNet18 to ResNet50)ResNet18 student · ImageNet valtop-1 error (%): 30.57 -> 30.95Hurt
early-stop the teacher training (WRN16-8, 20/65) (ablation)WRN16-1 student KD · CIFAR10test error (%): 8.028 -> 7.482Improved
early-stop the teacher training (WRN16-3, 20/65) (ablation)WRN16-1 student KD · CIFAR10test error (%): 7.681 -> 7.498Improved
early-stop the teacher training (ResNet152, 35 epochs) with ESKD (ablation)ResNet18 student · ImageNet valtop-1 error (%): 29.45 -> 29.26Improved
add attention transfer to ESKD (ResNet34 teacher)ResNet18 student · ImageNet valtop-1 error (%): 29.16 -> 28.84Improved
add early-stopped teacher (50 epochs) to ESKD + AT (ablation)ResNet18 student · ImageNet valtop-1 error (%): 28.84 -> 28.61Improved
add an intermediate medium teacher (Large->Med->Small) to distill to the small student (ablation)WRN16-1 student · CIFAR10test error (%): 7.614 -> 8.04Hurt
train the teacher with 5 sequential KD iterations instead of from scratch (ablation)WRN16-3 teacher for WRN16-1 · CIFAR10student test error (%): 7.61 -> 7.79No clear effect
ensemble sequential KD generations instead of ensemble of scratch modelsResNet8 · CIFAR10ensemble error (%): 10.176 -> 12.786Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy