AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels

Zhendong Yang, Ailing Zeng, Zhe Li, Tianke Zhang, Chun Yuan, Yu Li

ICCV 2023 · Label-efficient and robust learning › Knowledge distillation · approach: Distillation with causal and adaptive guidance · 176 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add normalized non-target loss (NKD) on top of target loss (ablation)KD, ResNet34 to ResNet18 · ImageNet valTop-1 accuracy (%): 71.06 -> 71.96Improved
use KD's unnormalized non-target loss instead of NKD's normalized one (ablation)KD, ResNet34 to ResNet18 · ImageNet valTop-1 accuracy (%): 71.96 -> 71.33Hurt
add target-class distillation alone (ablation)KD, ResNet34 to ResNet18 · ImageNet valTop-1 accuracy (%): 69.90 -> 71.06Improved
replace KD with NKD lossResNet18 on ImageNet · ImageNet valTop-1 accuracy (%): 71.03 -> 71.96Improved
combine target and non-target distillation in USKD (ablation)MobileNet · ImageNet valTop-1 accuracy (%): 70.18 -> 70.38Improved
apply USKD to ViT-like models via an extra linear layer on a middle layerDeiT-Tiny, DeiT-Small, Swin-Tiny, RegNetX-1.6GF · ImageNet valTop-1 accuracy (%): DeiT-Tiny 74.42 -> 74.97Improved
use USKD-trained ResNet-50 as backbone for Mask R-CNNMask R-CNN · COCO val2017AP / AR (box): +0.3 mAP, +0.5 mARImproved
add USKD self-distillation during trainingMobileNet-V2 · ImageNet training timeminutes per epoch: 10.17 -> 10.18No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy