Research map / Label-efficient and robust learning / Knowledge distillation
Rethinking distillation objectives
Revisits distillation losses via label smoothing, probability mass allocation and empirical analysis.
160 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Knowledge distillation, keyword cluster “teacher student · kd · teachers”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Leveling off Several papers from 2025–26 still report on 1 tracked benchmark, but the best result has barely moved since 2024.
25 papers in 2023–24 and 89 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 17.6% to 42.0%.
What looks settled
- CIFAR100 (accuracy): best up to 2024 82.08, best in 2025–26 82.02 from 2 papers, at most 0.2% better, if at all.
No longer reported by papers here: CIFAR100 (ResNet56→ResNet20) (top-1 accuracy, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning (CVPR 2017 · 1,664 citations)
- 2018 · Deep Mutual Learning (CVPR 2018 · 1,819 citations)
- 2019 · Variational Information Distillation for Knowledge Transfer (CVPR 2019 · 669 citations)
- 2020 · Improved Knowledge Distillation via Teacher Assistant (AAAI 2020 · 1,092 citations)
- 2021 · Cross-Layer Distillation with Semantic Calibration (AAAI 2021 · 311 citations)
- 2022 · Anomaly Detection via Reverse Distillation From One-Class Embedding (CVPR 2022 · 859 citations)
- 2023 · Learning To Retain While Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation (CVPR 2023 · 33 citations)
- 2024 · Scaled Decoupled Distillation (CVPR 2024 · 47 citations)
- 2025 · Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition (AAAI 2025 · 23 citations)
- 2026 · Distillation Dynamics: Towards Understanding Feature-Based Distillation in Vision Transformers (AAAI 2026 · 2 citations)
Earliest papers here
- A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning (CVPR 2017 · 1,664 citations)
- Deep Mutual Learning (CVPR 2018 · 1,819 citations)
- Knowledge Transfer with Jacobian Matching (ICML 2018 · 86 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
CIFAR100 · accuracy (higher is better)
8 papers, 2020–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 82.08 | Revisiting Knowledge Distillation via Label Smoothing Regularization (CVPR 2020 · 551 citations) |
CIFAR-100 · Top-1 accuracy (%) (higher is better)
4 papers, 2018–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 80.28 | Deep Mutual Learning (CVPR 2018 · 1,819 citations) |
CIFAR100 (ResNet56→ResNet20) · top-1 accuracy (higher is better)
4 papers, 2021–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 73.21 | Complementary Relation Contrastive Distillation (CVPR 2021 · 110 citations) |
ImageNet validation · Top-1 accuracy (%) (higher is better)
4 papers, 2022–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 72.41 | Knowledge Distillation via the Target-Aware Transformer (CVPR 2022 · 138 citations) |
| 2024 | 73.08 | Scaled Decoupled Distillation (CVPR 2024 · 47 citations) |
What has been tried
91 papers here report 1,043 measured changes (a component added, removed or swapped, compared with a baseline): helped 73%, no clear effect 6%, hurt 18%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add consistent decoupled logits | CIFAR-100 · accuracy | 73.33 -> 75.14 | Helped | Scaled Decoupled Distillation (CVPR 2024 · 47 citations) |
| Use teacher performance features, logits, and cross-entropy as the state input | CIFAR-100 · Acc | 1.01% gain over AVER; 77.84 | Helped | Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition (AAAI 2025 · 23 citations) |
| Replace KL-Div with WD for logit distillation without target/non-target separation | ImageNet, ResNet34 → ResNet18 · Top-1 accuracy | 71.03 -> 72.04 | Helped | Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation (NeurIPS 2024 · 21 citations) |
| Remove MGFD | WN18RR, 12 layers · Hits@1 | 58.8 -> 59.3 | Helped | Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph Completion (AAAI 2024 · 8 citations) |
| Add absolute-difference OT with token-wise ranking | QED, labeled distillation · F1 | 55.71 -> 56.76; 59.03 -> 59.71; 60.48 -> 61.22 | Helped | Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models (AAAI 2025 · 5 citations) |
| Add target-guided training to multiple-layer Kdiffusion | CIFAR-100, ResNet32x4 teacher and ShuffleNetV1 student · accuracy | 76.83 -> 77.26 | Helped | Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · 5 citations) |
| Add Co-Progression and Knowledge Prototype to FD | MVTec AD · image-level AUC-ROC | 95.63 -> 98.39 | Helped | Co-Progression Knowledge Distillation with Knowledge Prototype for Industrial Anomaly Detection (AAAI 2025 · 4 citations) |
| Learn an adaptive categorical distribution instead of using a uniform distribution | GLUE-dev, average across seven tasks · average score | 81.4 -> 82.1 | Helped | How to Trade Off the Quantity and Capacity of Teacher Ensemble: Learning Categorical Distribution to Stochastically Employ a Teacher for Distillation (AAAI 2024 · 4 citations) |
| Add the intra-sample STG relation | CIFAR-100 · Top-1 accuracy (%) | 73.92 -> 75.28 | Helped | Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · 2 citations) |
| Add feature blending (Lblend) to the GAN discriminator | CIFAR10 and CIFAR100 · accuracy | 92.87 (↓1.87); 74.18 (↓2.75) without Lblend | Helped | Hybrid Data-Free Knowledge Distillation (AAAI 2025 · 1 citation) |
| replace KL divergence with MSE | CIFAR10-DVS · accuracy | 78.93 -> 76.50; 79.33 -> 76.67 | Hurt | Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers (NeurIPS 2025 · 1 citation) |
| add constrained inter-angle and intra-angle diversity losses | CIFAR-100, RN32×4 teacher and RN8×4 student · accuracy | 75.46 -> 76.46 | Helped | Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 49 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-100 · 11 of 49 papers
- ImageNet · 8 of 49 papers
- CIFAR-10 · 7 of 49 papers
- TinyImageNet · 4 of 49 papers
- DAPO-Math-17k · 3 of 49 papers
- Cityscapes · 2 of 49 papers
- UltraChat 200k · 2 of 49 papers
- GSM8K · 2 of 49 papers
Baselines
- DKD · 8 of 49 papers
- KD · 8 of 49 papers
- SFT · 5 of 49 papers
- AT · 4 of 49 papers
- GRPO · 4 of 49 papers
- RKD · 3 of 49 papers
- SeqKD · 3 of 49 papers
- ReviewKD · 3 of 49 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 114 papers; from OpenAlex and the papers' affiliations).
- Google · 6 papers
- Chinese Academy of Sciences · 5 papers
- Korea Advanced Institute of Science and Technology · 4 papers
- University of Chinese Academy of Sciences · 3 papers
- Beihang University · 3 papers
- Tsinghua University · 3 papers
- Peking University · 3 papers
- Meituan · 2 papers
- IBM · 2 papers
- Tencent · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Distillation with causal and adaptive guidance (82 papers)
- Few-shot and metric distillation (80 papers)
- Teacher properties and robustness (54 papers)
- Knowledge distillation for detectors (25 papers)
Elsewhere in the topic
- Patient consistent distillation (68 papers)
- Self and data-free distillation (57 papers)
- Distillation for machine translation (18 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).