Research map / Label-efficient and robust learning / Knowledge distillation
Hybrid Data-Free Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HiDFD, a data-free distillation method that trains a teacher-guided ADCGAN on very few collected real examples and distills a student on a hybrid of inflated real and synthetic data using classifier-sharing feature alignment.
Problem
Data-free knowledge distillation performs poorly when collection-based methods lack enough real data and generation-based methods produce low-quality synthetic data; the paper asks whether a small set of collected data plus GAN-generated examples can train reliable students.
Approach
Guide an auxiliary-classifier GAN with the teacher network (feature blending and transfer losses against discriminator overfitting, class-frequency smoothing against generator imbalance), then inflate the few collected examples and share the teacher classifier while aligning student features to the teacher on the hybrid data.
Key results
- With only rho=0.1 collected data, HiDFD achieves CIFAR10 accuracy of 94.74% (ResNet34-ResNet18) versus 95.70% trained on full original data, and 76.93% on CIFAR100 versus 78.05%.
- On ImageNet (ResNet34-ResNet18, rho=1.0), HiDFD reaches 65.36% at rho=0.1 and 66.89% at rho=1.0 compared to the best prior baseline KD3 at 55.05%... (see Table 1 values for other methods).
- Ablation on CIFAR10 shows removing all teacher-guided generation losses drops accuracy by 5.19 points, and replacing classifier-sharing alignment with traditional KD methods lowers accuracy by 1.25 to 2.34 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add feature blending (Lblend) to the GAN discriminator (ablation) | HiDFD teacher-guided generation · CIFAR10 and CIFAR100 | accuracy: 92.87 (↓1.87); 74.18 (↓2.75) without Lblend | Helped |
| Add feature transfer (Ltrans) to the GAN discriminator (ablation) | HiDFD teacher-guided generation · CIFAR10 and CIFAR100 | accuracy: 91.86 (↓2.88); 74.40 (↓2.53) without Ltrans | Helped |
| Add category-frequency regularization (Lreg) to generator training (ablation) | HiDFD teacher-guided generation · CIFAR10 and CIFAR100 | accuracy: 92.76 (↓1.98); 73.95 (↓2.98) without Lreg | Helped |
| Add both feature blending and feature transfer (ablation) | HiDFD teacher-guided generation · CIFAR10 and CIFAR100 | accuracy: 91.10 (↓3.64); 71.02 (↓5.91) without Lblend, Ltrans | Helped |
| Add feature blending and category-frequency regularization (ablation) | HiDFD teacher-guided generation · CIFAR10 and CIFAR100 | accuracy: 90.77 (↓3.97); 71.83 (↓5.10) without Lblend, Lreg | Helped |
| Add feature transfer and category-frequency regularization (ablation) | HiDFD teacher-guided generation · CIFAR10 and CIFAR100 | accuracy: 91.42 (↓3.32); 72.10 (↓4.83) without Ltrans, Lreg | Helped |
| Add feature blending, feature transfer, and category-frequency regularization (ablation) | HiDFD teacher-guided generation · CIFAR10 and CIFAR100 | accuracy: 89.55 (↓5.19); 70.25 (↓6.68) without Lblend, Ltrans, Lreg | Helped |
| Use classifier-sharing feature alignment instead of OFAKD (ablation) | HiDFD student distillation · CIFAR10 and CIFAR100 | accuracy: 92.88 (↓1.86); 70.86 (↓6.07) with OFAKD | Helped |
| Use classifier-sharing feature alignment instead of VKD (ablation) | HiDFD student distillation · CIFAR10 and CIFAR100 | accuracy: 92.69 (↓2.05); 66.96 (↓9.97) with VKD | Helped |
| Use classifier-sharing feature alignment instead of SemcKD (ablation) | HiDFD student distillation · CIFAR10 and CIFAR100 | accuracy: 93.49 (↓1.25); 70.93 (↓6.00) with SemcKD | Helped |
| Use classifier-sharing feature alignment instead of CC (ablation) | HiDFD student distillation · CIFAR10 and CIFAR100 | accuracy: 92.63 (↓2.11); 69.52 (↓7.41) with CC | Helped |
| Use classifier-sharing feature alignment instead of DKD (ablation) | HiDFD student distillation · CIFAR10 and CIFAR100 | accuracy: 92.95 (↓1.79); 68.25 (↓8.68) with DKD | Helped |
Surprising findings
- The method using 120x less collected data than collection-based baselines reaches comparable accuracy to students trained on full original data on several benchmarks.
Limitations
- Effectiveness depends on the quality and representativeness of the collected data; biased collections propagate to synthetic data and student.
- Focuses only on classification tasks; applicability to detection or segmentation not tested.
Experimental setup
Datasets: CIFAR10 CIFAR100 CINIC TinyImageNet HAM ImageNet ImageNet subset (collected for CIFAR) WebVision ISIC
Benchmarks: CIFAR10 CIFAR100 CINIC TinyImageNet HAM ImageNet
Metrics: top-1 accuracy
Baselines: DAFL DDAD DI PRE CMI SSNet DeGAN DFND KD3 OFAKD VKD SemcKD
Builds on: ADCGAN DeGAN Data-free knowledge distillation (DAFL/DFND-style generation) KD3
Closest papers by meaning
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.93)
- Learning Student Networks in the Wild (CVPR 2021 · similarity 0.91)
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.91)
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.90)
- Learning To Retain While Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation (CVPR 2023 · similarity 0.90)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.