Research map / Label-efficient and robust learning / Knowledge distillation
Self and data-free distillation
Uses self-distillation, data-free settings, decoupled losses and ensemble distillation for efficient students.
57 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Knowledge distillation, keyword cluster “distilled · ensemble · synthetic”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Quiet None of the tracked benchmarks has a report from 2025–26.
28 papers in 2023–24 and 14 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 19.7% to 6.6%.
No longer reported by papers here: CIFAR-10 (accuracy, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Knowledge Distillation by On-the-Fly Native Ensemble (NeurIPS 2018 · 366 citations)
- 2019 · Zero-Shot Knowledge Distillation in Deep Networks (ICML 2019 · 82 citations)
- 2020 · Self-Distillation Amplifies Regularization in Hilbert Space (NeurIPS 2020 · 128 citations)
- 2021 · Peer Collaborative Learning for Online Knowledge Distillation (AAAI 2021 · 133 citations)
- 2022 · Decoupled Knowledge Distillation (CVPR 2022 · 960 citations)
- 2023 · Curriculum Temperature for Knowledge Distillation (AAAI 2023 · 229 citations)
- 2024 · Understanding the Role of the Projector in Knowledge Distillation (AAAI 2024 · 37 citations)
- 2025 · A Knowledge Distillation-Based Approach to Enhance Transparency of Classifier Models (AAAI 2025 · 5 citations)
- 2026 · Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · 0 citations)
Earliest papers here
- Knowledge Distillation by On-the-Fly Native Ensemble (NeurIPS 2018 · 366 citations)
- KDGAN: Knowledge Distillation with Generative Adversarial Networks (NeurIPS 2018 · 132 citations)
- Zero-Shot Knowledge Distillation in Deep Networks (ICML 2019 · 82 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
CIFAR-10 · accuracy (higher is better)
4 papers, 2023–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 95.01 | Generalization Matters: Loss Minima Flattening via Parameter Hybridization for Efficient Online Knowledge Distillation (CVPR 2023 · 16 citations) |
| 2024 | 95.44 | De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts (CVPR 2024 · 10 citations) |
What has been tried
54 papers here report 607 measured changes (a component added, removed or swapped, compared with a baseline): helped 70%, no clear effect 7%, hurt 17%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use batch normalization instead of L2 normalization | ImageNet-1K 20% subset, RegNet → MBv2 · top-1 accuracy | 52.91 -> 56.09 | Helped | Understanding the Role of the Projector in Knowledge Distillation (AAAI 2024 · 37 citations) |
| Increase the pre-selected subset size beyond N × IPC | ImageNet-10, ImageNet-100, ImageNet-1K · top-1 accuracy | Mixed | On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm (CVPR 2024 · 31 citations) | |
| Add priority sampling and diversity/difficulty balancing | CIFAR-10 · top-1 accuracy (%) | 87.35 -> 92.92 | Helped | Small Scale Data-Free Knowledge Distillation (CVPR 2024 · 21 citations) |
| replace Original PGD with Partial-PGD | CIFAR-10 · time (s) | 18.19 -> 4.37 | Helped | Layer Attack Unlearning: Fast and Accurate Machine Unlearning via Layer Level Attack and Knowledge Distillation (AAAI 2024 · 19 citations) |
| Use label-text embeddings instead of random noise or one-hot labels as generator input | CIFAR10 · accuracy | 90.14 -> 93.48 | Helped | NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillation (CVPR 2024 · 18 citations) |
| add fine-grained distillation at passage level | MS-Marco Doc Dev · MRR@100 | 0.435 -> 0.440 | Helped | Fine-Grained Distillation for Long Document Retrieval (AAAI 2024 · 12 citations) |
| Use learned attention weights instead of random λᵢ | CIFAR-10 and CIFAR-100 · accuracy | 93.92 vs 94.56; 74.79 vs 75.10 (Fast); 91.56 vs 93.67; 72.72 vs 74.59 (DeepInv); 95.28 vs 95.38; 76.86 vs 77.09 (DFND) | Helped | De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts (CVPR 2024 · 10 citations) |
| Add random masking | Argoverse 1 · minADE (K=1) | 1.4733 -> 1.4189 | Helped | LaKD: Length-agnostic Knowledge Distillation for Trajectory Prediction with Any Length Observations (NeurIPS 2024 · 8 citations) |
| Remove class-prior from the standard DFKD framework | CIFAR-100, five teacher models · peak accuracy | Teacher 1: 73.67 -> 73.96; Teacher 2: 71.09 -> 71.23; Teacher 3: 74.22 -> 74.43; Teacher 4: 73.14 -> 76.08; Teacher 5: 7 | Helped | Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · 6 citations) |
| vary α and T across the reported settings | Brain tumor test · accuracy | 0.9572; 0.9656; 0.9748; 0.9702; 0.9748; 0.9717 | Mixed | A Knowledge Distillation-Based Approach to Enhance Transparency of Classifier Models (AAAI 2025 · 5 citations) |
| Lower the intra-class temperature T | CIFAR-100 · classification accuracy | Helped | Debiased Distillation for Consistency Regularization (AAAI 2025 · 5 citations) | |
| add multi-label logits distillation (MLD) | MS-COCO · mAP | 70.31 -> 70.68 | Helped | Multi-label Self Knowledge Distillation (AAAI 2025 · 4 citations) |
What reviewers will expect
Datasets and baselines named in the 13 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-100 · 8 of 13 papers
- CIFAR-10 · 7 of 13 papers
- ImageNet-1K · 6 of 13 papers
- Tiny-ImageNet · 5 of 13 papers
- SVHN · 3 of 13 papers
- ImageNet · 2 of 13 papers
Baselines
- MTT · 4 of 13 papers
- SRe2L · 4 of 13 papers
- DSA · 4 of 13 papers
- DM · 4 of 13 papers
- DC · 3 of 13 papers
- KD · 2 of 13 papers
- DKD · 2 of 13 papers
- DOT · 2 of 13 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 42 papers; from OpenAlex and the papers' affiliations).
- Microsoft · 4 papers
- Zhejiang University · 4 papers
- Google DeepMind · 4 papers
- National University of Singapore · 3 papers
- Massachusetts Institute of Technology · 3 papers
- Tencent · 2 papers
- Beijing Institute of Technology · 2 papers
- Shanghai Jiao Tong University · 2 papers
- Xidian University · 2 papers
- Tsinghua University · 2 papers
Use it
Download BibTeX of 36 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Patient consistent distillation (68 papers)
- Distillation for machine translation (18 papers)
Elsewhere in the topic
- Rethinking distillation objectives (160 papers)
- Distillation with causal and adaptive guidance (82 papers)
- Few-shot and metric distillation (80 papers)
- Teacher properties and robustness (54 papers)
- Knowledge distillation for detectors (25 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).