Research map / Label-efficient and robust learning / Knowledge distillation
Few-shot and metric distillation
Distills compact students with limited data using grafting, regression and metric learning.
80 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Knowledge distillation, keyword cluster “teacher student · kd · teachers”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
15 papers in 2023–24 and 33 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 10.6% to 15.6%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Face Model Compression by Distilling Knowledge from Neurons (AAAI 2016 · 219 citations)
- 2018 · DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer (AAAI 2018 · 223 citations)
- 2019 · MEAL: Multi-Model Ensemble via Adversarial Learning (AAAI 2019 · 140 citations)
- 2020 · Self-Training With Noisy Student Improves ImageNet Classification (CVPR 2020 · 2,328 citations)
- 2021 · Knowledge distillation via softmax regression representation learning (ICLR 2021 · 55 citations)
- 2022 · Learning Multiple Adverse Weather Removal via Two-Stage Knowledge Learning and Multi-Contrastive Regularization: Toward a Unified Model (CVPR 2022 · 240 citations)
- 2023 · Masked Autoencoders Enable Efficient Knowledge Distillers (CVPR 2023 · 57 citations)
- 2024 · CLIP-KD: An Empirical Study of CLIP Model Distillation (CVPR 2024 · 65 citations)
- 2025 · Reinforced Multi-teacher Knowledge Distillation for Efficient General Image Forgery Detection and Localization (AAAI 2025 · 10 citations)
- 2026 · Revisiting Cross-Architecture Distillation: Adaptive Dual-Teacher Transfer for Lightweight Video Models (AAAI 2026 · 1 citation)
Earliest papers here
- Face Model Compression by Distilling Knowledge from Neurons (AAAI 2016 · 219 citations)
- DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer (AAAI 2018 · 223 citations)
- Paraphrasing Complex Network: Network Compression via Factor Transfer (NeurIPS 2018 · 86 citations)
What has been tried
54 papers here report 638 measured changes (a component added, removed or swapped, compared with a baseline): helped 77%, no clear effect 6%, hurt 14%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add feature distillation using MSE | ImageNet-1K · accuracy | 30.6 -> 34.2 | Helped | CLIP-KD: An Empirical Study of CLIP Model Distillation (CVPR 2024 · 65 citations) |
| increase generator depth from 1 to 2 | SSV2 · Top-1 accuracy | 72.06 -> 72.38 | Helped | Asymmetric Masked Distillation for Pre-Training Small Foundation Models (CVPR 2024 · 12 citations) |
| Add teacher knowledge distillation to Cue-Net | Multi-tampering · Detection F1 | 0.764 -> 0.778 | Helped | Reinforced Multi-teacher Knowledge Distillation for Efficient General Image Forgery Detection and Localization (AAAI 2025 · 10 citations) |
| add IDT with complete denoising | Reentrancy vulnerability detection · accuracy | 72.55 -> 90.98 | Helped | MTVHunter: Smart Contracts Vulnerability Detection Based on Multi-Teacher Knowledge Translation (AAAI 2025 · 5 citations) |
| Add L ˜ft alongside Lft | CIFAR-100 (400 labels) · error rate | 48.58 -> 32.51 | Helped | CustomKD: Customizing Large Vision Foundation for Edge Model Improvement via Knowledge Distillation (CVPR 2025 · 5 citations) |
| Add MFM pseudo labels to sparse-label training | DAIR-V2X · AP@0.5 | 49.49 -> 66.90 | Helped | CoDTS: Enhancing Sparsely Supervised Collaborative Perception with a Dual Teacher-Student Framework (AAAI 2025 · 2 citations) |
| Add dynamic loss-based domain sampling | Met and GLDv2; other five domains · R@1 | 9.4 -> 15.9; 33.4 -> 40.9; 62.9 -> 62.2; 72.5 -> 71.2; 82.2 -> 80.2; 90.8 -> 88.5; 88.6 -> 88.0 | Mixed | UDON: Universal Dynamic Online distillatioN for generic image representations (NeurIPS 2024 · 2 citations) |
| Distill from multiple teachers (8) instead of 1 or 2 | TDC regression and classification · average R2 / AUROC | regression avg R2 0.405 (1 teacher) -> 0.476 (8 teachers) | Helped | Learning Task-Agnostic Representations through Multi-Teacher Distillation (NeurIPS 2025 · 0 citations) |
| train on ImageNet-Merge rather than ImageNet-1K | 12 fine-grained datasets · average accuracy | 85.7 -> 87.8 | Helped | Accessing Vision Foundation Models via ImageNet-1K (ICLR 2025 · 0 citations) |
| Add Teacher-Specific Adapter Paths to preserve teacher representation biases | PASCAL-Context · representation similarity | Helped | Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning (ICLR 2025 · 0 citations) | |
| add reverse distillation to direct alignment | CAMO · average mIoU | 68.83 -> 70.22 | Helped | Mutual Learning for SAM Adaptation: A Dual Collaborative Network Framework for Source-Free Domain Transfer (ICML 2025) |
| add score-based distillation to direct student training | NQ dev reranking · R@1 | 39.5 -> 42.4 | Helped | USTAD: Unified Single-model Training Achieving Diverse Scores for Information Retrieval (ICML 2024) |
What reviewers will expect
Datasets and baselines named in the 17 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet-1K · 3 of 17 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 48 papers; from OpenAlex and the papers' affiliations).
- Peking University · 3 papers
- Zhejiang University · 2 papers
- Shenzhen University · 2 papers
- Google · 2 papers
- Microsoft · 2 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Rethinking distillation objectives (160 papers)
- Distillation with causal and adaptive guidance (82 papers)
- Teacher properties and robustness (54 papers)
- Knowledge distillation for detectors (25 papers)
Elsewhere in the topic
- Patient consistent distillation (68 papers)
- Self and data-free distillation (57 papers)
- Distillation for machine translation (18 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).