Research map / Label-efficient and robust learning / Knowledge distillation
Cross-Architecture Knowledge Distillation
Yufan Liu, Jiajiong Cao, Bing Li, Weiming Hu, Jingting Ding, Liang Li, Stephen J. Maybank
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill from a ViT-L/16 Transformer teacher with the proposed projectors instead of Logits KD | ResNet50 student · ImageNet | accuracy: 77.45 -> 78.85 | Improved |
| distill from Swin-L teacher with the proposed method instead of Logits KD | ResNet50x2 student · ImageNet | accuracy: 79.68 -> 81.39 | Improved |
| use proposed cross-architecture KD on CIFAR100 instead of Logits | ResNet50x2 student, Swin-L teacher · CIFAR100 | accuracy: 88.93 -> 91.21 | Improved |
| use ViT-L/16 teacher instead of ViT-B/16 teacher (ablation) | ResNet50 student · ImageNet | accuracy: 78.34 -> 78.85 | Improved |
| use Swin-L teacher instead of ViT-B/16 teacher (ablation) | EfficientNetB0 student · ImageNet | accuracy: 79.23 -> 79.52 | Improved |
| add the PCA and GL projectors (ablation) | cross-architecture KD · ImageNet | accuracy | Improved |
| add cross-view robust training (ablation) | student networks · ImageNet (regular evaluation) | accuracy: 0.2%-0.4% | Improved |
| add cross-view robust training (ablation) | student networks · ImageNet (noisy evaluation) | accuracy: more than 1.0% | Improved |
| apply cross-architecture KD to detection with ViT-L/16 teacher instead of Logits | ResNet50 · COCO detection | AP: 34.9 -> 35.5 | Improved |
| use ViT-L/16 teacher Logits KD instead of ResNet152x2 teacher Logits KD | ResNet50 · COCO detection | AP: 35.0 -> 34.9 | Hurt |
| apply cross-architecture KD instead of Logits | ResNet18 · CelebA-Spoof face anti-spoofing | EER: 1.6 -> 1.3 | Improved |
| use Transformer teacher with proposed KD instead of CNN-CNN KD | ResNet50x2 CIFAR100 · CIFAR100 | accuracy: 90.04 -> 90.97 | Improved |
Closest papers by meaning
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.86)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.86)
- Understanding the Role of the Projector in Knowledge Distillation (AAAI 2024 · similarity 0.86)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.86)
- Distilling Knowledge from Heterogeneous Architectures for Semantic Segmentation (AAAI 2025 · similarity 0.86)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
