Research map / Label-efficient and robust learning / Knowledge distillation
Coupling the Generator with Teacher for Effective Data-Free Knowledge Distillation
Xu Chen, Yang Li, Yahong Han, Guangquan Xu, Jialie Shen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add feature-level alignment loss L_align to the generator (ablation) | CPNet · CIFAR-100 (ResNet34->ResNet18) | top-1 accuracy (%): 61.32 -> 76.56 | Improved |
| add feature-level adversarial loss L_adv (ablation) | CPNet · CIFAR-10 (ResNet34->ResNet18) | top-1 accuracy (%): 94.56 -> 94.63 | Improved |
| add feature MSE distillation loss at penultimate layer (ablation) | CPNet · CIFAR-10 (ResNet34->ResNet18) | top-1 accuracy (%): 94.63 -> 94.66 | No clear effect |
| use coupled generator instead of CMI generator (ablation) | CPNet · CIFAR-10 (ResNet34->ResNet18) | top-1 accuracy (%): 93.84 -> 94.48 | Improved |
| use coupled generator instead of noisy-layer generator (ablation) | CPNet · CIFAR-100 (ResNet34->ResNet18) | top-1 accuracy (%): 76.29 -> 76.45 | Improved |
| split teacher into 4 blocks instead of 1 or 8 (ablation) | CPNet · CIFAR-10 (ResNet34->ResNet18) | top-1 accuracy (%): 87.26 -> 94.66 (1 to 4 blocks); 94.66 -> 90.01 (4 to 8 blocks) | Mixed |
| CPNet-300 vs NAYER-300 with same schedule | NAYER · TinyImageNet (ResNet34->ResNet18) | top-1 accuracy (%): 64.17 -> 64.91 | Improved |
| use CPNet-100 for training efficiency | DFKD pipeline · CIFAR-10 (WRN40-2->WRN16-2) | training time (hours): 1.80 -> 1.39 (vs NAYER-100) | Improved |
Closest papers by meaning
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.90)
- Hybrid Data-Free Knowledge Distillation (AAAI 2025 · similarity 0.89)
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.89)
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.89)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.87)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
