AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Distilling the Undistillable: Learning from a Nasty Teacher

Surgan Jandial, Yash Khasbage, Arghya Pal, Vineeth N Balasubramanian, Balaji Krishnamurthy

ECCV 2022 · Label-efficient and robust learning › Knowledge distillation · approach: Distillation with causal and adaptive guidance · 6 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use high-temperature softened teacher output composed with one-hot (HTC) instead of plain Nasty KDNasty KD · CIFAR-100 ResNet18->MobileNetV2student accuracy: 3.15 -> 71.01Improved
use sequence of contrastive models ensemble (SCM) instead of Nasty KDNasty KD · TinyImageNet ResNet18->MobileNetV2student accuracy: 0.85 -> 61.01Improved
use SCM instead of SkepticalSkeptical · TinyImageNet ResNet18->MobileNetV2student accuracy: 47.37 -> 61.01Improved
use SCM sequence length 4 instead of 2 (ablation)SCM · CIFAR-100 ShuffleNetV2student accuracy: 74.45 -> 75.23Improved
use SCM sequence length 4 instead of 2 (ablation)SCM · CIFAR-100 CNNstudent accuracy: 60.99 -> 61.31Improved
generate SCM sequence with ShuffleNetV2 instead of ResNet18 architecture (ablation)SCM · CIFAR-100 ShuffleNetV2student accuracy: 63.67 -> 72.71Improved
replace Nasty teacher by SCM ensemble (ablation)Nasty Teacher · CIFAR-100 ResNet18KL divergence to original teacher: 14.6363 -> 5.8398Improved
apply HTC when no labels are availableNasty KD · CIFAR-100 ShuffleNetV2student accuracy: 64.41 -> 74.17Improved
apply HTC in data-free distillationNasty KD · CIFAR-10 MobileNetV2student accuracy: 41.79 -> 52.96Improved
use only 10% of the data with HTCNasty KD · CIFARstudent accuracyImproved
use intermediate SCM sequence model as teacher (improved defense)Nasty KD · CIFAR-10 MobileNetV2student accuracy: 31.73 -> 22.12Improved
use early-stopped Nasty teacher matched in accuracy instead of SCM-based defense (ablation)Ours defense · CIFAR-10 ShuffleNetV2student accuracy: 73.23 -> 87.81Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy