AI Paper Atlas

Research map / Label-efficient and robust learning

Knowledge distillation: research map

783 papers on Knowledge distillation in Label-efficient and robust learning, from ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR (2016–2026), grouped into 2 clusters and 8 approaches. The busiest year so far is 2026.

What it is

Knowledge distillation trains a small "student" model to imitate a large "teacher" model, so the student gets much of the teacher's accuracy at a fraction of the cost. Research looks at what to imitate (outputs, features, relations), when distillation fails, and how to distill without the original data.

Terms you will see on this page:

Teacher and student
the large trained model and the smaller model learning from it.
Soft labels
the teacher's full probability distribution, richer than one-hot labels.
Data-free distillation
distilling with synthetic inputs when the training data is unavailable.

Start reading with the most cited papers here:

Within Label-efficient and robust learning, its share grew from 4.9% in 2023–24 to 6.9% in 2025–26 (142 → 212 papers at ICML, NeurIPS, ICLR, CVPR and AAAI, the venues with data for all four years).

2016: 1162017: 4172018: 14182019: 35192020: 58202021: 91212022: 90222023: 110232024: 105242025: 111252026: 16426

Explore Knowledge distillation in the interactive map

Working on something in this topic? Describe your idea in scime atlas to see which approach it falls under, the closest papers by meaning and how crowded the spot has become.

What changed recently

Approaches whose share of the topic moved most between 2023–24 and 2025–26 (ICML, NeurIPS, ICLR, CVPR and AAAI):

Papers from 2025–2026 that are already cited most:

Approaches and key papers

Teacher-student distillation · 623 papers
Keywords: teacher student, kd, teachers

Approaches in this cluster:

Most cited and most cited since 2024:

Self-distillation, ensembles and synthetic data · 160 papers
Keywords: distilled, ensemble, synthetic

Approaches in this cluster:

Most cited and most cited since 2024:

Related topics in Label-efficient and robust learning

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy