AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Self-supervised Models are Good Teaching Assistants for Vision Transformers

Haiyan Wu, Yuting Gao, Yinqi Zhang, Shaohui Lin, Yuan Xie, Xing Sun, Ke Li

ICML 2022 · Label-efficient and robust learning › Knowledge distillation · approach: Few-shot and metric distillation

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add a self-supervised teaching assistant (SSTA, DINO) alongside SL teacher with head-level attention distillationDeiT-Ti · ImageNet-1KAcc@1: +1.8 (72.2 to 74.0)Improved
add SSTA on top of DeiT-Ti⚗ (RegNetY-16G hard-label distillation)DeiT-Ti⚗ · ImageNet-1KAcc@1: +0.7 (74.5 to 75.2)Improved
add SSTA head-level distillationXCiT-T12 · ImageNet-1KAcc@1: +0.5 (77.0 to 77.5)Improved
use a second SL teacher (2SL) instead of SSL teacher with early stopping (ablation)DeiT-Ti · ImageNet-1KAcc@1: 73.2 vs 73.2 for single SL early100No clear effect
add SSL teacher (SSTA) to SL teacher versus two SL teachers, both early-stopped at 100 epochs (ablation)DeiT-Ti · ImageNet-1KAcc@1: +0.8 (73.2 to 74.0)Improved
distill from SSL teacher alone with early stop vs SL teacher alone (ablation)DeiT-Ti · ImageNet-1KAcc@1: 72.6 vs 73.2Hurt
remove early stopping (distill for all 300 epochs) in SSTA (ablation)DeiT-Ti · ImageNet-1KAcc@1: 72.2 vs 74.0 with early stop at 100 epochsHurt
select most important head by accuracy drop instead of averaging all heads (ablation)SSTA distillation on DeiT-Ti · ImageNet-1KAcc@1: +0.8 (73.2 to 74.0)Improved
select most important head instead of random head selection (ablation)SSTA distillation on DeiT-Ti · ImageNet-1KAcc@1: +0.5 (73.5 to 74.0)Improved
early stop distillation at 100 epochs (vs 50, 150, 200, 250, or never) (ablation)SSTA distillation on DeiT-Ti · ImageNet-1KAcc@1: 74.0 at 100 vs 73.4 (50), 73.4 (150), 73.7 (200), 72.6 (250), 72.2 (300)Improved
distill on layers {10,11,12} instead of {12}, {11,12}, or {9,10,11,12} (ablation)SSTA distillation on DeiT-Ti · ImageNet-1KAcc@1: 74.0 vs 73.1 ({12}), 73.4 ({11,12}), 73.6 ({9-12})Improved
use attention transfer (AT) or logits KD instead of head-level distillation alongside SSTA (ablation)DeiT-Ti · ImageNet-1KAcc@1: LKD 73.4 and AT 70.0 vs head-level 74.0; AT is below baseline 72.2Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy