AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Knowledge Distillation: A Survey

Jianping Gou, Baosheng Yu, Stephen J. Maybank, Dacheng Tao

IJCV 2021 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 4,385 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
distill from a ResNet56 teacher with feature knowledge (FT)ResNet20 · CIFAR10accuracy: 92.22 -> 93.15Improved
distill with relation knowledge (FSP)ResNet8 · CIFAR10accuracy: 87.91 -> 88.70Improved
distill from ResNet152 with TOFDShuffleNetV2 · CIFAR100accuracy: 72.38 -> 76.68Improved
distill from ResNet152 with TOFDResNet50 · CIFAR100accuracy: 77.42 -> 84.74Improved
distill from ResNet50 with SSKDMobileNetV2 · CIFAR100accuracy: 65.79 -> 72.57Improved
online distillation with Rocket-KDWRN-16-1 · CIFAR100accuracy: 56.30 -> 67.00Improved
online mutual learning (DML) with MobileNet peerResNet32 · CIFAR100accuracy: 68.99 -> 76.13Improved
add online ensemble teacher (ONE)ResNet32 · CIFAR10accuracy: 93.07 -> 94.01Improved
add attention-based online distillation (ACNs)AlexNet · CIFAR100accuracy: 39.45 -> 46.27Improved
add self-distillation (SD)ResNet32 · CIFAR100accuracy: 68.39 -> 71.29Improved
add Tf-KD self-distillationResNeXt29 · CIFAR100accuracy: 81.03 -> 82.08Improved
add self-distillation with response and feature knowledgeDenseNet40 · CIFAR10accuracy: 94.53 -> 94.80Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy