Research map / Label-efficient and robust learning / Knowledge distillation
KDGAN: Knowledge Distillation with Generative Adversarial Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add teacher with mutual distillation to two-player GAN (KDGAN vs NaGAN) (ablation) | NaGAN · CIFAR-10 n=500 | accuracy: 46.29 -> 57.56 | Helped |
| add teacher with mutual distillation to two-player GAN (ablation) | NaGAN · MNIST n=100 | training epochs to converge: about 135 -> about 25 epochs | Helped |
| add discriminator to KD (KDGAN vs best KD baseline CODIS) | KD classifier · MNIST n=100 | accuracy: 74.02 -> 77.95 | Helped |
| add Gumbel-Max concrete relaxation (ablation) | KDGAN · MNIST n=100 | accuracy: 0.7605 -> 0.7795 | Helped |
| add Gumbel-Max concrete relaxation (ablation) | KDGAN · MNIST n=100 | training epochs: around 45% fewer epochs | Helped |
| use adversarial-only training (NaGAN) instead of KD with small data | classifier · CIFAR-10 n=500 | accuracy: 54.17 -> 46.29 | Hurt |
| use adversarial-only training (NaGAN) instead of KD with large data | classifier · CIFAR-10 n=50,000 | accuracy: 85.12 -> 85.34 | Helped |
| set distillation weight beta very small (0.0001) (ablation) | KDGAN · MNIST n=100 | accuracy: more than 2% drop | Hurt |
| vary alpha and beta (ablation) | KDGAN · MNIST | accuracy | No clear effect |
| large teacher-from-classifier weight gamma (ablation) | KDGAN · MNIST | accuracy | Hurt |
Closest papers by meaning
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.89)
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.89)
- Hybrid Data-Free Knowledge Distillation (AAAI 2025 · similarity 0.89)
- Robust Distillation via Untargeted and Targeted Intermediate Adversarial Samples (CVPR 2024 · similarity 0.87)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.87)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.