Research map / Label-efficient and robust learning / Knowledge distillation
Zero-shot Knowledge Transfer via Adversarial Belief Matching
Paul Micaelli, Amos J. Storkey
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adversarially generated pseudo-images to train the student against the teacher, without training data | WRN-40-2 teacher and WRN-16-1 student · CIFAR-10 | accuracy: 83.69 \mypm 0.58\% | Improved |
| Fine-tune the zero-shot student with 100 real images per class | Zero-shot KT · CIFAR-10, 100 images per class | accuracy: 83.69 \mypm 0.58\% -> 85.91 \mypm 0.24 \% | Improved |
| Use zero-shot adversarial belief matching instead of KD+AT distillation | WRN-40-2 teacher and WRN-16-1 student · SVHN | MTE: 0.64 -> 0.09 | Improved |
| Use zero-shot adversarial belief matching instead of KD+AT distillation | WRN-40-2 teacher and WRN-16-1 student · CIFAR-10 | MTE: 0.68 -> 0.22 | Improved |
| Add attention transfer to the student loss (ablation) | Zero-shot KT · CIFAR-10 and SVHN, across architectures | accuracy: 2\% | Improved |
| Replace forward KL with reverse KL (ablation) | Zero-shot KT · Student test performance | accuracy | Hurt |
| Replace forward KL with Jensen–Shannon divergence (ablation) | Zero-shot KT · Student test performance | accuracy | Hurt |
| Add a teacher-entropy objective to the generator loss (ablation) | Zero-shot KT · Student test performance | accuracy | Hurt |
| Add a student-entropy objective to the generator loss (ablation) | Zero-shot KT · Student test performance | accuracy | Hurt |
| Add image-consistency regularization to the generator loss (ablation) | Zero-shot KT · Student test performance | accuracy | Hurt |
| Add diversity regularization to the generator loss (ablation) | Zero-shot KT · Student test performance | accuracy | Hurt |
Closest papers by meaning
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.88)
- Hybrid Data-Free Knowledge Distillation (AAAI 2025 · similarity 0.88)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.86)
- Adversarial Moment-Matching Distillation of Large Language Models (NeurIPS 2024 · similarity 0.86)
- When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need (ICML 2025 · similarity 0.86)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
