Research map / Label-efficient and robust learning / Knowledge distillation
Distillation with causal and adaptive guidance
Improves distillation using causal intervention, dual guidance and out-of-domain data.
82 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Knowledge distillation, keyword cluster “teacher student · kd · teachers”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 2 of 4 tracked benchmarks.
25 papers in 2023–24 and 35 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 17.6% to 16.5%.
Open questions the data points to
- CIFAR-100 (accuracy): best went from 78.88 (up to 2024) to 81.6 in 2025 — All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · 1 citation)
- CIFAR-10 (accuracy): best went from 93.41 (up to 2024) to 98.14 in 2025 — What Makes a Good Dataset for Knowledge Distillation? (CVPR 2025 · 0 citations)
No longer reported by papers here: CIFAR-100, ResNet-50 → MobileNetV2 (Top-1 accuracy (%), last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Born Again Neural Networks (ICML 2018 · 431 citations)
- 2019 · Relational Knowledge Distillation (CVPR 2019 · 1,698 citations)
- 2020 · Online Knowledge Distillation with Diverse Peers (AAAI 2020 · 309 citations)
- 2021 · Wasserstein Contrastive Representation Distillation (CVPR 2021 · 92 citations)
- 2022 · Evaluation-Oriented Knowledge Distillation for Deep Face Recognition (CVPR 2022 · 42 citations)
- 2023 · Multi-Level Logit Distillation (CVPR 2023 · 139 citations)
- 2024 · Logit Standardization in Knowledge Distillation (CVPR 2024 · 243 citations)
- 2025 · Unlocking the Potential of Reverse Distillation for Anomaly Detection (AAAI 2025 · 17 citations)
- 2026 · S^2-KD: Semantic-Spectral Knowledge Distillation Spatiotemporal Forecasting (AAAI 2026 · 4 citations)
Earliest papers here
- Born Again Neural Networks (ICML 2018 · 431 citations)
- Relational Knowledge Distillation (CVPR 2019 · 1,698 citations)
- Online Knowledge Distillation with Diverse Peers (AAAI 2020 · 309 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
CIFAR-100 · accuracy (higher is better)
7 papers, 2021–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 78.88 | Undistillable: Making A Nasty Teacher That CANNOT teach students (ICLR 2021 · 19 citations) |
| 2025 | 81.6 | All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · 1 citation) |
ImageNet, ResNet-34 → ResNet-18 · Top-1 accuracy (higher is better)
6 papers, 2022–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 71.88 | Respecting Transfer Gap in Knowledge Distillation (NeurIPS 2022 · 8 citations) |
| 2023 | 72.17 | KD-Zero: Evolving Knowledge Distiller for Any Teacher-Student Pairs (NeurIPS 2023 · 17 citations) |
| 2024 | 72.19 | Knowledge Distillation Based on Transformed Teacher Matching (ICLR 2024 · 5 citations) |
CIFAR-10 · accuracy (higher is better)
5 papers, 2021–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 93.41 | Undistillable: Making A Nasty Teacher That CANNOT teach students (ICLR 2021 · 19 citations) |
| 2025 | 98.14 | What Makes a Good Dataset for Knowledge Distillation? (CVPR 2025 · 0 citations) |
CIFAR-100, ResNet-50 → MobileNetV2 · Top-1 accuracy (%) (higher is better)
4 papers, 2022–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 71.84 | Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation (NeurIPS 2022 · 3 citations) |
What has been tried
59 papers here report 663 measured changes (a component added, removed or swapped, compared with a baseline): helped 71%, no clear effect 5%, hurt 19%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use full Z-score standardization instead of original logits | CIFAR-100 val · top-1 accuracy | 73.60 -> 74.14; 74.38 -> 76.11; 74.45 -> 76.56; 73.33 -> 76.62; 68.29 -> 76.56; 65.34 -> 76.61; 63.45 -> 76.33 | Helped | Logit Standardization in Knowledge Distillation (CVPR 2024 · 243 citations) |
| add Frequency Prompt to low-frequency distillation | COCO · AP | 40.7 -> 41.0 | Helped | FreeKD: Knowledge Distillation via Semantic Frequency Prompt (CVPR 2024 · 40 citations) |
| Add Expert | MVTec AD · P-AUC | 98.02 -> 98.69 | Helped | Unlocking the Potential of Reverse Distillation for Anomaly Detection (AAAI 2025 · 17 citations) |
| Add aleatory uncertainty-aware contrastive learning | AS2009 · AUC | 0.8507±0.0020 -> 0.8563±0.0018 | Helped | Uncertainty-aware Knowledge Tracing (AAAI 2025 · 12 citations) |
| add bias elimination | CIFAR-100, ResNet-50 → ResNet-18 · accuracy | 71.56 -> 79.70 | Helped | Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · 6 citations) |
| Add Lkd to the baseline | Cityscapes val, both distillation modes · mIoU | 0.94% average mIoU gain | Helped | Distilling Knowledge from Heterogeneous Architectures for Semantic Segmentation (AAAI 2025 · 6 citations) |
| Add DCR and DMM to the student distillation framework | Urban100 ×2 · PSNR | 32.67 -> 32.92 | Helped | Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration (AAAI 2025 · 6 citations) |
| Add pre-trained teacher guidance alongside BOR guidance | CIFAR-100; ResNet32×4 → ResNet8×4 · Top-1 accuracy | 76.45 -> 77.44 | Helped | Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · 5 citations) |
| Remove similar counterfactuals using geometric set-cover | FashionMNIST, median across 10 models · accuracy | 79% -> 83% | Helped | Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · 5 citations) |
| Add sample-adaptive weighting to TTM to form WTTM | CIFAR-100, WRN-40-2 → WRN-16-2 · Top-1 accuracy (%) | 76.23 -> 76.37 | Helped | Knowledge Distillation Based on Transformed Teacher Matching (ICLR 2024 · 5 citations) |
| Use on-policy student-generated sequences instead of fixed supervised sequences | XSum · ROUGE-2 | Helped | On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (ICLR 2024 · 3 citations) | |
| Remove NC2 (ETF matching) loss from NCKD | CIFAR-100 ResNet-8x4 student · top-1 accuracy | 77.23 full vs 75.98 without NC2 on ResNet-8x4 (-1.25) | Helped | Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · 2 citations) |
What reviewers will expect
Datasets and baselines named in the 17 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-100 · 7 of 17 papers
- ImageNet-1K · 5 of 17 papers
- ImageNet · 3 of 17 papers
- MS-COCO · 3 of 17 papers
- Tiny-ImageNet · 3 of 17 papers
- CIFAR-10 · 2 of 17 papers
- GSM8K · 2 of 17 papers
Baselines
- KD · 5 of 17 papers
- DKD · 5 of 17 papers
- FitNet · 4 of 17 papers
- CRD · 4 of 17 papers
- CTKD · 3 of 17 papers
- DIST · 3 of 17 papers
- RKD · 3 of 17 papers
- SeqKD · 3 of 17 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 60 papers; from OpenAlex and the papers' affiliations).
- Huawei · 4 papers
- East China Normal University · 3 papers
- Chinese Academy of Sciences · 3 papers
- The University of Sydney · 3 papers
- Zhejiang University · 3 papers
- Peking University · 3 papers
- Nanjing University · 2 papers
- The Ohio State University · 2 papers
- Beihang University · 2 papers
- National University of Defense Technology · 2 papers
Use it
Download BibTeX of 37 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Rethinking distillation objectives (160 papers)
- Few-shot and metric distillation (80 papers)
- Teacher properties and robustness (54 papers)
- Knowledge distillation for detectors (25 papers)
Elsewhere in the topic
- Patient consistent distillation (68 papers)
- Self and data-free distillation (57 papers)
- Distillation for machine translation (18 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).