Research map / Label-efficient and robust learning
Knowledge distillation: research map
783 papers on Knowledge distillation in Label-efficient and robust learning, from ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR (2016–2026), grouped into 2 clusters and 8 approaches. The busiest year so far is 2026.
What it is
Knowledge distillation trains a small "student" model to imitate a large "teacher" model, so the student gets much of the teacher's accuracy at a fraction of the cost. Research looks at what to imitate (outputs, features, relations), when distillation fails, and how to distill without the original data.
Terms you will see on this page:
- Teacher and student
- the large trained model and the smaller model learning from it.
- Soft labels
- the teacher's full probability distribution, richer than one-hot labels.
- Data-free distillation
- distilling with synthetic inputs when the training data is unavailable.
Start reading with the most cited papers here:
- Self-Training With Noisy Student Improves ImageNet Classification (CVPR 2020 · 2,328 citations)
- Deep Mutual Learning (CVPR 2018 · 1,819 citations)
Within Label-efficient and robust learning, its share grew from 4.9% in 2023–24 to 6.9% in 2025–26 (142 → 212 papers at ICML, NeurIPS, ICLR, CVPR and AAAI, the venues with data for all four years).
Explore Knowledge distillation in the interactive map
Working on something in this topic? Describe your idea in scime atlas to see which approach it falls under, the closest papers by meaning and how crowded the spot has become.
What changed recently
Approaches whose share of the topic moved most between 2023–24 and 2025–26 (ICML, NeurIPS, ICLR, CVPR and AAAI):
- Rethinking distillation objectives rising: 18% → 42% of the topic (25 → 89 papers)
- Few-shot and metric distillation rising: 11% → 16% of the topic (15 → 33 papers)
- Self and data-free distillation falling: 20% → 7% of the topic (28 → 14 papers)
- Distillation for machine translation falling: 8% → 1% of the topic (11 → 2 papers)
Papers from 2025–2026 that are already cited most:
- Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition (AAAI 2025 · 23 citations)
Approaches and key papers
Teacher-student distillation · 623 papers
Keywords: teacher student, kd, teachers
Approaches in this cluster:
- Rethinking distillation objectives (216 papers)
Revisits distillation losses via label smoothing, probability mass allocation and empirical analysis. - Distillation with causal and adaptive guidance (127 papers)
Improves distillation using causal intervention, dual guidance and out-of-domain data. - Few-shot and metric distillation (155 papers)
Distills compact students with limited data using grafting, regression and metric learning. - Teacher properties and robustness (60 papers)
Studies what makes teachers effective, including adversarial, random and noisy-input teachers. - Knowledge distillation for detectors (65 papers)
Distill detectors by mimicking features, rankings, and focal or global knowledge.
Most cited and most cited since 2024:
- Self-Training With Noisy Student Improves ImageNet Classification (CVPR 2020 · 2,328 citations)
- Deep Mutual Learning (CVPR 2018 · 1,819 citations)
- Logit Standardization in Knowledge Distillation (CVPR 2024 · 243 citations)
- CLIP-KD: An Empirical Study of CLIP Model Distillation (CVPR 2024 · 65 citations)
Self-distillation, ensembles and synthetic data · 160 papers
Keywords: distilled, ensemble, synthetic
Approaches in this cluster:
- Patient consistent distillation (77 papers)
Distills with consistent teachers, relational knowledge and soft labels, including dataset distillation. - Self and data-free distillation (65 papers)
Uses self-distillation, data-free settings, decoupled losses and ensemble distillation for efficient students. - Distillation for machine translation (18 papers)
Applies selective and uncertainty-aware distillation to non-autoregressive and simultaneous translation.
Most cited and most cited since 2024:
- Decoupled Knowledge Distillation (CVPR 2022 · 960 citations)
- When does label smoothing help? (NeurIPS 2019 · 879 citations)
- Decoupled Kullback-Leibler Divergence Loss (NeurIPS 2024 · 41 citations)
- Understanding the Role of the Projector in Knowledge Distillation (AAAI 2024 · 37 citations)
Related topics in Label-efficient and robust learning
- Noisy labels (576)
- Zero-/few-shot and meta-learning (1,029)
- Test-time adaptation (437)
- OOD and anomaly detection (836)
- Recognition and long-tail (6,306)
- Semi-supervised learning (897)
- Self-supervised and contrastive (1,267)
- Domain adaptation and generalization (2,012)
