AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Zero-shot Knowledge Transfer via Adversarial Belief Matching

Paul Micaelli, Amos J. Storkey

NeurIPS 2019 · Label-efficient and robust learning › Knowledge distillation · approach: Teacher properties and robustness · 33 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use adversarially generated pseudo-images to train the student against the teacher, without training dataWRN-40-2 teacher and WRN-16-1 student · CIFAR-10accuracy: 83.69 \mypm 0.58\%Improved
Fine-tune the zero-shot student with 100 real images per classZero-shot KT · CIFAR-10, 100 images per classaccuracy: 83.69 \mypm 0.58\% -> 85.91 \mypm 0.24 \%Improved
Use zero-shot adversarial belief matching instead of KD+AT distillationWRN-40-2 teacher and WRN-16-1 student · SVHNMTE: 0.64 -> 0.09Improved
Use zero-shot adversarial belief matching instead of KD+AT distillationWRN-40-2 teacher and WRN-16-1 student · CIFAR-10MTE: 0.68 -> 0.22Improved
Add attention transfer to the student loss (ablation)Zero-shot KT · CIFAR-10 and SVHN, across architecturesaccuracy: 2\%Improved
Replace forward KL with reverse KL (ablation)Zero-shot KT · Student test performanceaccuracyHurt
Replace forward KL with Jensen–Shannon divergence (ablation)Zero-shot KT · Student test performanceaccuracyHurt
Add a teacher-entropy objective to the generator loss (ablation)Zero-shot KT · Student test performanceaccuracyHurt
Add a student-entropy objective to the generator loss (ablation)Zero-shot KT · Student test performanceaccuracyHurt
Add image-consistency regularization to the generator loss (ablation)Zero-shot KT · Student test performanceaccuracyHurt
Add diversity regularization to the generator loss (ablation)Zero-shot KT · Student test performanceaccuracyHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy