Research map / Label-efficient and robust learning / Knowledge distillation
Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies data-free knowledge distillation from a teacher pretrained on imbalanced data, showing adversarial DFKD harms majority classes, and proposes IPAD, a primal-dual class-adaptive constrained objective for the generator.
Problem
Data-free knowledge distillation from a teacher pretrained on class-imbalanced data: adversarial DFKD favors minority classes and causes mode collapse on majority classes, degrading overall student accuracy.
Approach
Adds per-class KL constraints relative to the mean KL divergence to the adversarial generator objective, solved via Lagrangian primal-dual updates, so the generator does not over-focus on minority-class synthetic samples; requires no knowledge of the original class distribution.
Key results
- On CIFAR-100 (r=0.01) with ResNet34->ResNet18, IPAD reaches 54.9% overall accuracy versus 52.5% for ABM and 52.9% for MAD.
- On Tiny-ImageNet (r=0.2) with ResNet34->ResNet18, IPAD achieves 53.2% overall, close to the student's train-from-scratch 53.7%.
- Vanilla adversarial DFKD drops many-shot accuracy by 8.2 points relative to the teacher on imbalanced CIFAR-100 (r=0.01) while few-shot accuracy slightly increases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IPAD instead of ABM | DFKD, ResNet34 → ResNet18 · CIFAR-100 (r = 0.01) | accuracy: 52.5 -> 54.9 | Helped |
| Use IPAD instead of MAD | DFKD, WRN40-2 → WRN40-1 · CIFAR-100 (r = 0.01) | accuracy: 48.1 -> 49.7 | Helped |
| Use IPAD instead of CMI | DFKD, WRN40-2 → WRN16-2 · CIFAR-100 (r = 0.01) | accuracy: 50.7 -> 52.2 | Helped |
| Use IPAD instead of CMI | DFKD, ResNet34 → VGG16 · CIFAR-100 (r = 0.01) | accuracy: 47.2 -> 49.2 | Helped |
| Use IPAD instead of CMI | DFKD, ResNet34 → ResNet18 · Tiny-ImageNet (r = 0.2) | accuracy: 50.7 -> 53.2 | Helped |
| Use IPAD instead of MAD | DFKD, WRN40-2 → WRN40-1 · Tiny-ImageNet (r = 0.2) | accuracy: 33.5 -> 36.0 | Helped |
| Use IPAD instead of MAD | DFKD, WRN40-2 → WRN16-2 · Tiny-ImageNet (r = 0.2) | accuracy: 37.2 -> 38.6 | Helped |
| Use IPAD instead of MAD | DFKD, ResNet34 → VGG16 · Tiny-ImageNet (r = 0.2) | accuracy: 40.6 -> 43.2 | Helped |
| Add class-adaptive constraints on class-wise KL divergence | IPAD · CIFAR-100 (r = 0.01), WRN40-2 → WRN16-2 | accuracy: 50.4 -> 52.2 | Helped |
Surprising findings
- Adversarial DFKD from a biased teacher improves few-shot class accuracy while degrading majority-class accuracy.
- Non-adversarial DAFL underperforms adversarial baselines in most imbalanced settings, and fails badly on Tiny-ImageNet.
Limitations
- Theoretical analysis is limited to a binary Gaussian mixture with a linear classifier.
- Baselines were augmented with an entropy-maximization term for fair comparison, which may not reflect their original designs.
Experimental setup
Datasets: CIFAR-100 Tiny-ImageNet Food101 Places365 ImageNet
Benchmarks: CIFAR-100 Tiny-ImageNet
Metrics: top-1 accuracy many/medium/few shot accuracy
Baselines: DAFL ABM CMI MAD
Builds on: Adversarial Belief Matching CMI MAD DAFL two-stage imbalanced training (Kang et al. 2019)
Closest papers by meaning
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.92)
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.92)
- Hybrid Data-Free Knowledge Distillation (AAAI 2025 · similarity 0.91)
- Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation (AAAI 2024 · similarity 0.90)
- Robust Distillation via Untargeted and Targeted Intermediate Adversarial Samples (CVPR 2024 · similarity 0.90)
- When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.