Research map / Label-efficient and robust learning / Knowledge distillation
Self-supervised Models are Good Teaching Assistants for Vision Transformers
Haiyan Wu, Yuting Gao, Yinqi Zhang, Shaohui Lin, Yuan Xie, Xing Sun, Ke Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a self-supervised teaching assistant (SSTA, DINO) alongside SL teacher with head-level attention distillation | DeiT-Ti · ImageNet-1K | Acc@1: +1.8 (72.2 to 74.0) | Improved |
| add SSTA on top of DeiT-Ti⚗ (RegNetY-16G hard-label distillation) | DeiT-Ti⚗ · ImageNet-1K | Acc@1: +0.7 (74.5 to 75.2) | Improved |
| add SSTA head-level distillation | XCiT-T12 · ImageNet-1K | Acc@1: +0.5 (77.0 to 77.5) | Improved |
| use a second SL teacher (2SL) instead of SSL teacher with early stopping (ablation) | DeiT-Ti · ImageNet-1K | Acc@1: 73.2 vs 73.2 for single SL early100 | No clear effect |
| add SSL teacher (SSTA) to SL teacher versus two SL teachers, both early-stopped at 100 epochs (ablation) | DeiT-Ti · ImageNet-1K | Acc@1: +0.8 (73.2 to 74.0) | Improved |
| distill from SSL teacher alone with early stop vs SL teacher alone (ablation) | DeiT-Ti · ImageNet-1K | Acc@1: 72.6 vs 73.2 | Hurt |
| remove early stopping (distill for all 300 epochs) in SSTA (ablation) | DeiT-Ti · ImageNet-1K | Acc@1: 72.2 vs 74.0 with early stop at 100 epochs | Hurt |
| select most important head by accuracy drop instead of averaging all heads (ablation) | SSTA distillation on DeiT-Ti · ImageNet-1K | Acc@1: +0.8 (73.2 to 74.0) | Improved |
| select most important head instead of random head selection (ablation) | SSTA distillation on DeiT-Ti · ImageNet-1K | Acc@1: +0.5 (73.5 to 74.0) | Improved |
| early stop distillation at 100 epochs (vs 50, 150, 200, 250, or never) (ablation) | SSTA distillation on DeiT-Ti · ImageNet-1K | Acc@1: 74.0 at 100 vs 73.4 (50), 73.4 (150), 73.7 (200), 72.6 (250), 72.2 (300) | Improved |
| distill on layers {10,11,12} instead of {12}, {11,12}, or {9,10,11,12} (ablation) | SSTA distillation on DeiT-Ti · ImageNet-1K | Acc@1: 74.0 vs 73.1 ({12}), 73.4 ({11,12}), 73.6 ({9-12}) | Improved |
| use attention transfer (AT) or logits KD instead of head-level distillation alongside SSTA (ablation) | DeiT-Ti · ImageNet-1K | Acc@1: LKD 73.4 and AT 70.0 vs head-level 74.0; AT is below baseline 72.2 | Mixed |
Closest papers by meaning
- Teaching Matters: Investigating the Role of Supervision in Vision Transformers (CVPR 2023 · similarity 0.90)
- An Empirical Study of Training Self-Supervised Vision Transformers (ICCV 2021 · similarity 0.90)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.87)
- Attention Transfer Is Not Universally Effective for Vision Transformers (NeurIPS 2026 · similarity 0.86)
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers (ICLR 2025 · similarity 0.86)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
