Research map / Label-efficient and robust learning / Knowledge distillation
What Makes a Good Dataset for Knowledge Distillation?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies which surrogate datasets (real ID/OOD, synthetic OpenGL shaders, Leaves, noise) can distill a teacher when original data is unavailable, and identifies criteria (balanced teacher prediction histogram, diversity, decision-boundary info) plus an adversarial pair-based sampling attack.
Problem
Knowledge distillation typically assumes the teacher's original training data is available, but this often fails (continual learning, proprietary training data), so practitioners need to know what surrogate data works for distillation.
Approach
Shows that unoptimized, unnatural synthetic OOD imagery (e.g., OpenGL shaders) can serve as a distillation dataset, and that success is governed by how evenly the teacher's argmax predictions cover classes (relative entropy) and how well decision boundaries are sampled; proposes a paired adversarial perturbation to improve sampling.
Key results
- Distilling a CIFAR10 ResNet50 teacher with OpenGL shader images reaches 94.02% vs 95.98% with original data for ResNet18 student.
- Distilling with OpenGL shaders improved a CIFAR10 student by 62.2% from worst to best configuration under augmentation changes, and extreme augmentation plus mixup with CIFAR10 matched OpenGL performance.
- The adversarial decision-boundary attack raised FGVCA-to-CIFAR10 distillation accuracy by 76.8 points (from 11.39 to 88.14 on worst datasets).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OpenGL shader images instead of the teacher's original CIFAR10 data | Vanilla KD, ResNet50 → ResNet18 · CIFAR10 | accuracy: 95.98 -> 94.02 | Hurt |
| Use OpenGL shader images instead of the teacher's original CIFAR100 data | Vanilla KD, ResNet50 → ResNet18 · CIFAR100 | accuracy: 78.35 -> 73.27 | Hurt |
| Use OpenGL shader images instead of the teacher's original EuroSAT data | Vanilla KD, ResNet50 → ResNet18 · EuroSAT | accuracy: 98.60 -> 98.45 | Hurt |
| Use OpenGL shader images with a more complex ConvNeXt-T teacher | Vanilla KD, ConvNeXt-T → ResNet18 · CIFAR10 | accuracy: 98.14 -> 56.43 | Hurt |
| Use OpenGL shader images with a more complex ConvNeXt-T teacher | Vanilla KD, ConvNeXt-T → ResNet18 · EuroSAT | accuracy: 98.85 -> 95.85 | Hurt |
| Replace temperature-scaled soft targets with one-hot targets (ablation) | Vanilla KD, ResNet50 → ResNet18 · CIFAR10 | accuracy: 95.98 -> 96.09 | No clear effect |
| Replace temperature-scaled soft targets with one-hot targets (ablation) | Vanilla KD, ResNet50 → ResNet18 · OpenGL shaders → CIFAR10 | accuracy: 94.02 -> 91.68 | Hurt |
| Replace temperature-scaled soft targets with label-smoothed targets (ablation) | Vanilla KD, ResNet50 → ResNet18 · OpenGL shaders → CIFAR10 | accuracy: 94.02 -> 92.07 | Hurt |
| Use balanced rather than long-tail 20K examples (ablation) | Vanilla KD without mixup, ResNet50 → ResNet18 · CIFAR10 | accuracy: 91.85 -> 93.12 | Helped |
| Add mixup to 20K long-tail and balanced examples (ablation) | Vanilla KD, ResNet50 → ResNet18 · CIFAR10 | accuracy: 91.85 -> 94.36; 93.12 -> 95.07 | Helped |
| Remove data augmentation while retaining mixup (ablation) | Vanilla KD, ResNet50 → ResNet18 · CIFAR10 | accuracy: 95.98 -> 95.00 | Hurt |
| Remove data augmentation while retaining mixup (ablation) | Vanilla KD, ResNet50 → ResNet18 · OpenGL shaders → CIFAR10 | accuracy: 93.39 -> 92.20 | Hurt |
Surprising findings
- Unoptimized unnatural synthetic OpenGL shader images can distill teachers nearly as well as real in-domain data.
- A surrogate real dataset (Pets ImageNet-ID subset vs original) can outperform the original, though it uses more samples.
Limitations
- Experiments use 400 epochs and small-scale image classification datasets; performance gaps may close with longer training not fully tested.
- Surrogate datasets underperform in sample efficiency relative to in-domain data, and fine-grained/many-class tasks struggle with synthetic data.
- Vanilla KD only; more advanced KD methods not studied.
Experimental setup
Datasets: CIFAR10 CIFAR100 Tiny ImageNet ImageNet FGVC-Aircraft Pets Food EuroSAT OpenGL shaders (TwiGL) Leaves noise
Benchmarks: CIFAR10 CIFAR100 Tiny ImageNet FGVC-Aircraft Pets EuroSAT
Metrics: test accuracy relative entropy of class prediction histogram
Baselines: Vanilla KD on original data CMI PRE-DFKD Spaceship FAST Single Image KD
Builds on: Knowledge distillation (Hinton et al.) Function matching KD (Beyer et al.) Data-free KD Mixup
Closest papers by meaning
- What is Dataset Distillation Learning? (ICML 2024 · similarity 0.89)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.88)
- De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts (CVPR 2024 · similarity 0.86)
- DD-Ranking: Rethinking the Evaluation of Dataset Distillation (NeurIPS 2026 · similarity 0.86)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.85)
- A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.