Research map / Architectures and efficiency / Attention and transformers

ScaleKD: Strong Vision Transformers Could Be Excellent Teachers

NeurIPS 2024 · Architectures and efficiency › Attention and transformers · approach: Efficient and robust vision transformers · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ScaleKD, a cross-architecture feature distillation method that uses a pre-trained ViT teacher (Swin-L, BEiT-L/14) to train CNN, MLP and ViT students on ImageNet-1K from scratch, approaching pre-training-level accuracy with far fewer viewed samples.

Problem

Transferring the scalable knowledge of large pre-trained ViTs to heterogeneous student architectures (CNN, MLP, ViT) under a setting where only ImageNet-1K is available, given differences in feature computing paradigm, model scale, and knowledge density.

Approach

Three coupled components: a cross attention projector (transformer decoder with trainable queries) to align computing paradigms; dual-view feature mimicking that adds a path matching teacher features with the DC (zero-frequency) component removed; and teacher parameter perception, a proxy path connecting the student's early stages to the teacher's last stage so student parameters align with teache

Key results

• With Swin-L as teacher, ScaleKD gains +3.05 to +5.52 top-1 on 8 IN-1K student backbones (e.g., ResNet-50 82.03%, ViT-B/16 85.53%), mean gain 3.94% over 11 teacher-student pairs.
• ResNet-50 trained by ScaleKD reaches 82.55% top-1 (600 epochs), beating FunMatch's 82.31% at 9600 epochs.
• ScaleKD uses up to 195x fewer viewed samples than cross-modal pre-training (e.g., ViT-B/14 at 86.43% vs CLIP-based 86.17%).

Measured effects reported in the paper

Surprising findings

• Student models trained from scratch with ScaleKD on IN-1K outperform the same architectures pre-trained on IN-22K or JFT-300M followed by IN-1K fine-tuning, without viewing any pre-training data.
• The ViT-S/16 student trained with ScaleKD (83.93%) outperforms ViT-S/16 pre-trained on IN-22K with KD (82.00%).

Limitations

• Experiments are not run on very large teachers such as ViT-22B or large students such as ViT-L due to compute limits.
• Training cost grows with teacher scale; TPP is the most resource-consuming component, especially combined with DFM.
• Requires a strong pre-trained ViT teacher, so the approach depends on the availability of such teachers.

Experimental setup

Datasets: ImageNet-1K MS-COCO ADE20K CIFAR-100

Benchmarks: ImageNet-1K MS-COCO ADE20K CIFAR-100

Metrics: top-1 accuracy AP mIoU FLOPs parameters GPU memory training time

Baselines: FD DIST DiffKD OFA FunMatch KD AT OFD RKD CRD DKD SRRL

Builds on: feature distillation DIST logits-based knowledge distillation Mask R-CNN UperNet

Closest papers by meaning

• Revisiting Cross-Architecture Distillation: Adaptive Dual-Teacher Transfer for Lightweight Video Models (AAAI 2026 · similarity 0.90)
• Scaling Vision Transformers (CVPR 2022 · similarity 0.90)
• Are Large Kernels Better Teachers than Transformers for ConvNets? (ICML 2023 · similarity 0.90)
• VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.89)
• On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.88)
• MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders (CVPR 2025 · similarity 0.88)