Research map / Label-efficient and robust learning / Knowledge distillation
Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TA-DFKD, which drops the class-prior loss from generator training and uses GMM-based teacher-driven sample selection to keep only clean synthetic samples, yielding robust data-free distillation across teacher models.
Problem
Data-free knowledge distillation (DFKD) is highly sensitive to the choice of teacher model, sometimes failing catastrophically even with well-trained teachers, because generators trained with class-prior and adversarial losses produce either low-diversity or low-quality samples.
Approach
Treat the teacher as a lenient expert: remove the class-prior restriction on the generator (to allow diversity) and instead filter synthetic samples with a two-component Gaussian Mixture Model over the teacher's per-sample cross-entropy loss, using only selected samples for the adversarial and KD losses.
Key results
- On CIFAR-100, TA-DFKD achieves the highest acc_last across five ResNet-34 teachers (e.g., 76.55±0.1 for Teacher 1 vs. 76.13 for PRE-DFKD), with smaller variance than baselines.
- On CIFAR-10 with five teachers, TA-DFKD reaches converging accuracies of 94.24, 91.99, 90.21, 93.61, 90.27 versus CMI's best of 92.54, 89.84, 89.40, 93.26, 89.62 on the same teachers, while DAFL shows a ±20.0 std on Teacher 3.
- On Tiny-ImageNet with two teachers, TA-DFKD reaches 53.00 and 53.55 converging accuracy, above DAFL (47.76, 50.60) and PRE-DFKD (46.45, 46.13).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TA-DFKD instead of the strongest listed baseline | ResNet-18 student distilled from ResNet-34 · CIFAR-10, five teacher models | peak accuracy: Teacher 1: 94.10 -> 94.43; Teacher 2: 90.16 -> 92.15; Teacher 3: 89.81 -> 90.69; Teacher 4: 93.61 -> 93.79; Teacher 5: 9 | Mixed |
| Use TA-DFKD instead of the strongest listed baseline | ResNet-18 student distilled from ResNet-34 · CIFAR-100, five teacher models | peak accuracy: Teacher 1: 76.57 -> 76.76; Teacher 2: 73.53 -> 73.89; Teacher 3: 75.44 -> 76.02; Teacher 4: 77.10 -> 76.99; Teacher 5: 7 | Mixed |
| Use TA-DFKD instead of the strongest listed baseline | ResNet-18 student distilled from ResNet-34 · Tiny-ImageNet, two teacher models | peak accuracy: Teacher 1: 53.16 -> 54.84; Teacher 2: 53.68 -> 54.52 | Helped |
| Remove class-prior from the standard DFKD framework (ablation) | DFKD student · CIFAR-100, five teacher models | peak accuracy: Teacher 1: 73.67 -> 73.96; Teacher 2: 71.09 -> 71.23; Teacher 3: 74.22 -> 74.43; Teacher 4: 73.14 -> 76.08; Teacher 5: 7 | Helped |
| Add teacher-driven sample selection to the no-class-prior variant (ablation) | DFKD without class-prior · CIFAR-100, five teacher models | peak accuracy: Teacher 1: 73.96 -> 76.76; Teacher 2: 71.23 -> 73.89; Teacher 3: 74.43 -> 76.02; Teacher 4: 76.08 -> 76.99; Teacher 5: 7 | Helped |
| Remove class-prior from the standard DFKD framework (ablation) | DFKD student · CIFAR-10, five teacher models | average accuracy: Teacher 1: 93.25 -> 93.06; Teacher 2: 90.54 -> 90.61; Teacher 3: 88.52 -> 88.71; Teacher 4: 92.88 -> 92.84; Teacher 5: 8 | Mixed |
| Add teacher-driven sample selection to the no-class-prior variant (ablation) | DFKD without class-prior · CIFAR-10, five teacher models | average accuracy: Teacher 1: 93.06 -> 94.35; Teacher 2: 90.61 -> 92.10; Teacher 3: 88.71 -> 90.46; Teacher 4: 92.84 -> 93.72; Teacher 5: 9 | Helped |
Surprising findings
- Removing the class-prior loss alone (w/o class-prior) gives little gain over baseline on CIFAR-100 for some teachers, and sample selection is needed to improve performance further.
- DAFL with a single teacher on CIFAR-10 sometimes collapses into a near-chance student (e.g., ±20.0 variance on Teacher 3), indicating class-prior-driven generators can collapse onto few easy samples.
Limitations
- Evaluated only on small image classification benchmarks (CIFAR-10/100, Tiny-ImageNet) with ResNet-34/ResNet-18 and DCGAN.
- CMI and DFAD could not be run on Tiny-ImageNet because proper hyperparameters were not found, so the Tiny-ImageNet comparison is limited to two baselines.
- Sample selection discards 10-40% of generated samples early in training, which adds a threshold hyperparameter tau and GMM fitting per batch.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet
Metrics: peak accuracy (acc_max) converging accuracy (acc_last[k]) FID standard deviation across runs
Baselines: DAFL DFAD CMI PRE-DFKD
Builds on: DAFL DFAD CMI PRE-DFKD DivideMix
Closest papers by meaning
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.92)
- When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need (ICML 2025 · similarity 0.91)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.91)
- Hybrid Data-Free Knowledge Distillation (AAAI 2025 · similarity 0.90)
- AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation (ICLR 2024 · similarity 0.90)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.