Research map / Label-efficient and robust learning / Knowledge distillation
Small Scale Data-Free Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSD-KD, a data-free knowledge distillation method that balances category diversity and sample difficulty in a small synthetic set and uses priority sampling from a replay buffer, reaching ~10x less synthetic data with much lower training time.
Problem
Adversarial data-free knowledge distillation requires generating very large synthetic sample sets, making end-to-end training costly; existing methods ignore the efficiency of the distillation stage.
Approach
Uses a modulating function that penalizes over-represented teacher-predicted classes and low-confidence easy samples (diversity- and difficulty-aware balancing), plus a priority sampling function with importance-sampling correction over a dynamic replay buffer, so distillation trains on a small, well-balanced set of synthetic samples.
Key results
- On CIFAR-100, SSD-KD with 5000 synthetic samples beats Fast2 by 3.29% to 10.19% absolute top-1 accuracy across 5 teacher-student pairs while training 1.5-2.5x faster (e.g., ResNet-34 teacher: 73.05% at 1.12h vs 69.76% at 4.02h).
- On NYUv2 semantic segmentation, SSD-KD reaches 0.384 mIoU with 16K synthetic samples, above Fast10 (0.366) and the student baseline (0.375).
- Ablation on synthetic data scale shows CIFAR accuracy drops by less than 10% even at 500 synthetic samples (1% of data).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SSD-KD instead of Fast10 | ResNet-34→ResNet-18 · CIFAR-10 | top-1 accuracy (%): 94.05 -> 94.26 | Helped |
| Use SSD-KD instead of Fast10 | VGG-11→ResNet-18 · CIFAR-10 | top-1 accuracy (%): 90.53 -> 90.67 | Helped |
| Use SSD-KD instead of Fast10 | WRN40-2→WRN16-1 · CIFAR-10 | top-1 accuracy (%): 89.29 -> 89.96 | Helped |
| Use SSD-KD instead of Fast10 | WRN40-2→WRN40-1 · CIFAR-10 | top-1 accuracy (%): 92.51 -> 93.23 | Helped |
| Use SSD-KD instead of Fast10 | WRN40-2→WRN16-2 · CIFAR-10 | top-1 accuracy (%): 92.45 -> 93.11 | Helped |
| Use SSD-KD instead of Fast10 | ResNet-34→ResNet-18 · CIFAR-100 | top-1 accuracy (%): 74.34 -> 75.16 | Helped |
| Use SSD-KD instead of Fast10 | VGG-11→ResNet-18 · CIFAR-100 | top-1 accuracy (%): 67.44 -> 68.77 | Helped |
| Use SSD-KD instead of Fast10 | WRN40-2→WRN16-1 · CIFAR-100 | top-1 accuracy (%): 54.02 -> 55.61 | Helped |
| Use SSD-KD instead of Fast10 | WRN40-2→WRN40-1 · CIFAR-100 | top-1 accuracy (%): 63.91 -> 64.57 | Helped |
| Use SSD-KD instead of Fast10 | WRN40-2→WRN16-2 · CIFAR-100 | top-1 accuracy (%): 65.12 -> 65.28 | Helped |
| Add priority sampling and diversity/difficulty balancing (ablation) | SSD-KD · CIFAR-10 | top-1 accuracy (%): 87.35 -> 92.92 | Helped |
| Add priority sampling and diversity/difficulty balancing (ablation) | SSD-KD · CIFAR-100 | top-1 accuracy (%): 64.05 -> 73.05 | Helped |
Surprising findings
- With small synthetic data scales (10% of training set), student models trained on synthetic samples outperform those trained on original samples.
Limitations
- Relaxing data scale to match Fast5 is needed for competitive accuracy on some pairs, trading efficiency for accuracy.
- Evaluated only on CIFAR-10/100 and NYUv2 with small teacher-student pairs.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 NYUv2
Benchmarks: CIFAR-10 CIFAR-100 NYUv2
Metrics: top-1 accuracy mIoU training time (hours)
Baselines: Fast2 Fast5 Fast10 DeepInv CMI DAFL DFQ ZSKT Vanilla KD DFND DFAD
Builds on: DeepInv Fast (meta-learned data-free KD) prioritized experience replay focal loss
Closest papers by meaning
- Hybrid Data-Free Knowledge Distillation (AAAI 2025 · similarity 0.93)
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.92)
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.91)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.91)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.89)
- Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.