AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Cross-Architecture Knowledge Distillation

Yufan Liu, Jiajiong Cao, Bing Li, Weiming Hu, Jingting Ding, Liang Li, Stephen J. Maybank

IJCV 2024 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 27 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
distill from a ViT-L/16 Transformer teacher with the proposed projectors instead of Logits KDResNet50 student · ImageNetaccuracy: 77.45 -> 78.85Improved
distill from Swin-L teacher with the proposed method instead of Logits KDResNet50x2 student · ImageNetaccuracy: 79.68 -> 81.39Improved
use proposed cross-architecture KD on CIFAR100 instead of LogitsResNet50x2 student, Swin-L teacher · CIFAR100accuracy: 88.93 -> 91.21Improved
use ViT-L/16 teacher instead of ViT-B/16 teacher (ablation)ResNet50 student · ImageNetaccuracy: 78.34 -> 78.85Improved
use Swin-L teacher instead of ViT-B/16 teacher (ablation)EfficientNetB0 student · ImageNetaccuracy: 79.23 -> 79.52Improved
add the PCA and GL projectors (ablation)cross-architecture KD · ImageNetaccuracyImproved
add cross-view robust training (ablation)student networks · ImageNet (regular evaluation)accuracy: 0.2%-0.4%Improved
add cross-view robust training (ablation)student networks · ImageNet (noisy evaluation)accuracy: more than 1.0%Improved
apply cross-architecture KD to detection with ViT-L/16 teacher instead of LogitsResNet50 · COCO detectionAP: 34.9 -> 35.5Improved
use ViT-L/16 teacher Logits KD instead of ResNet152x2 teacher Logits KDResNet50 · COCO detectionAP: 35.0 -> 34.9Hurt
apply cross-architecture KD instead of LogitsResNet18 · CelebA-Spoof face anti-spoofingEER: 1.6 -> 1.3Improved
use Transformer teacher with proposed KD instead of CNN-CNN KDResNet50x2 CIFAR100 · CIFAR100accuracy: 90.04 -> 90.97Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy