AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Analyzing the Confidentiality of Undistillable Teachers in Knowledge Distillation

Souvik Kundu, Qirui Sun, Yao Fu, Massoud Pedram, Peter Anthony Beerel

NeurIPS 2021 · Label-efficient and robust learning › Knowledge distillation · approach: Distillation with causal and adaptive guidance · 14 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
distill from nasty teacher at shallow auxiliary classifier with hybrid self-distillation (skeptical student)ResNet18 student, nasty ResNet50 teacher · CIFAR-100accuracy (%): 72.28 -> 77.25Improved
skeptical student for MobileNetV2 from nasty ResNet50MobileNetV2 student · CIFAR-100accuracy (%): 7.72 -> 66.38Improved
skeptical student on Tiny-ImageNet from nasty teacherResNet18 student · Tiny-ImageNetaccuracy (%): 53.60 -> 65.76Improved
skeptical student from a normal teacherResNet50 student · CIFAR-100accuracy (%): 79.56 -> 80.66Improved
skeptical student from a normal teacherResNet18 student · CIFAR-10 (ResNet50 teacher)accuracy (%): 95.43 -> 95.31No clear effect
ensemble ACs and final classifier when distilling from a nasty teacher (ablation)skeptical student MobileNetV2 · CIFAR-100accuracy (%): 66.38 -> 62.93Hurt
ensemble ACs and final classifier when distilling from a normal teacher (ablation)skeptical student ResNet50 · CIFAR-100accuracy (%): 80.66 -> 81.96Improved
distill from nasty teacher at the final classifier versus shallow AC1 (ablation)ResNet18 student · CIFAR-100accuracy (%): 77.19 -> 72.47Hurt
skeptical student for data-free KD (grey-box, nasty ResNet50)ResNet18 student · CIFAR-10 data-freeaccuracy (%): 80.34 -> 86.14Improved
skeptical student for data-free KD (black-box, no AT loss, nasty ResNet50)ResNet18 student · CIFAR-10 data-freeaccuracy (%): 20.95 -> 79.93Improved
skeptical (instead of normal) primary student as teacher for secondary studentResNet18 secondary student · CIFAR-100accuracy (%): 70.99 -> 79.22Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy