Research map / Label-efficient and robust learning / Knowledge distillation
Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation
Kien Do, Hung Le, Dung Nguyen, Dang Nguyen, HARIPRIYA HARIKUMAR, Truyen Tran, Santu Rana, Svetha Venkatesh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add EMA generator samples to student training (MAD vs ABM generator-only) | MAD · ImageNet (AlexNet) | accuracy (%): 41.23 -> 45.48 | Improved |
| add EMA generator samples to student training (vs ABM) | MAD · CIFAR100 (WRN) | accuracy (%): 62.59 -> 64.05 | Improved |
| add EMA generator samples to student training (vs ABM) | MAD · TinyImageNet | accuracy (%): 59.75 -> 62.32 | Improved |
| use EMA generator instead of memory bank of past synthetic samples (DFKD-Mem) | MAD · ImageNet (AlexNet) | accuracy (%): 43.30 -> 45.48 | Improved |
| use memory bank of past samples instead of generator-only (DFKD-Mem vs ABM) | ABM · CIFAR100 (WRN) | accuracy (%): 62.59 -> 61.25 | Hurt |
| set lambda_0=0 (drop samples from generator G, keep EMA only) (ablation) | MAD · CIFAR100 | accuracy | Hurt |
| reduce either lambda_0 or lambda_1 below 1 (ablation) | MAD · CIFAR100 | accuracy | Hurt |
| increase EMA momentum alpha to 1.0 (ablation) | MAD · CIFAR100 | accuracy: about 50% | Hurt |
| increase momentum alpha up to 0.95 (then too large at 0.999) (ablation) | MAD · CIFAR100 | accuracy | Mixed |
| increase number of student update steps (ablation) | MAD · CIFAR100 | accuracy | Improved |
| MAD vs CMI on WRN teacher/student | MAD · CIFAR100 (WRN) | accuracy (%): 68.75 -> 64.05 | Hurt |
Closest papers by meaning
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.92)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.91)
- Hybrid Data-Free Knowledge Distillation (AAAI 2025 · similarity 0.89)
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.89)
- Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation (AAAI 2024 · similarity 0.88)
- When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
