Research map / Label-efficient and robust learning / Knowledge distillation
Patient consistent distillation
Distills with consistent teachers, relational knowledge and soft labels, including dataset distillation.
68 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2019–2026) · an approach in Knowledge distillation, keyword cluster “distilled · ensemble · synthetic”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
23 papers in 2023–24 and 20 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 16.2% to 9.4%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2019 · When does label smoothing help? (NeurIPS 2019 · 879 citations)
- 2020 · Regularizing Class-Wise Predictions via Self-Knowledge Distillation (CVPR 2020 · 320 citations)
- 2021 · SuperMix: Supervising the Mixing Data Augmentation (CVPR 2021 · 106 citations)
- 2022 · Knowledge Distillation: A Good Teacher Is Patient and Consistent (CVPR 2022 · 241 citations)
- 2023 · MMANet: Margin-Aware Distillation and Modality-Aware Regularization for Incomplete Multimodal Learning (CVPR 2023 · 67 citations)
- 2024 · Decoupled Kullback-Leibler Divergence Loss (NeurIPS 2024 · 41 citations)
- 2025 · MergeNet: Knowledge Migration Across Heterogeneous Models, Tasks, and Modalities (AAAI 2025 · 5 citations)
- 2026 · Hard Labels In! Rethinking the Role of Hard Labels in Mitigating Local Semantic Drift (ICML 2026 · 0 citations)
Earliest papers here
- When does label smoothing help? (NeurIPS 2019 · 879 citations)
- Distilled Person Re-Identification: Towards a More Scalable System (CVPR 2019 · 111 citations)
- Unifying Heterogeneous Classifiers With Distillation (CVPR 2019 · 60 citations)
What has been tried
53 papers here report 627 measured changes (a component added, removed or swapped, compared with a baseline): helped 71%, no clear effect 6%, hurt 21%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| enable breaking asymmetric optimization (BA) | ImageNet, ResNet34 teacher to ResNet18 student · Top-1 accuracy (%) | 71.03 -> 71.80 | Helped | Decoupled Kullback-Leibler Divergence Loss (NeurIPS 2024 · 41 citations) |
| add entropy loss to safety-aware attention | Town05 Short Benchmark · Driving Score | 84.00 -> 93.69 | Helped | On the Road to Portability: Compressing End-to-End Motion Planner for Autonomous Driving (CVPR 2024 · 16 citations) |
| add local representation distillation (C1) | Cora, 3-shot · accuracy | 89.4(0.8) -> 92.8(0.8) | Helped | Self-Training Based Few-Shot Node Classification by Knowledge Distillation (AAAI 2024 · 11 citations) |
| add ID embedding disturbance | Deezer tail users (≤ T) · NDCG@10 | 0.3258 -> 0.3550 | Helped | Tail-STEAK: Improve Friend Recommendation for Tail Users via Self-Training Enhanced Knowledge Distillation (AAAI 2024 · 7 citations) |
| Add uncertainty estimation to pairwise margin weighting | iVQA · Acc@1 | 41.9 -> 41.5 | Helped | Ranking Distillation for Open-Ended Video Question Answering with Insufficient Labels (CVPR 2024 · 7 citations) |
| Add self-learning between knowledge-transfer phases (set T_cycle to 4) | CIFAR-100 · performance | 60.09 -> 66.51 | Helped | MergeNet: Knowledge Migration Across Heterogeneous Models, Tasks, and Modalities (AAAI 2025 · 5 citations) |
| scale the decoder from PLD-Tiny to PLD-Large | AVG11 (Real) · word accuracy | 90.08 -> 91.67 | Helped | Accurate Scene Text Recognition with Efficient Model Scaling and Cloze Self-Distillation (CVPR 2025 · 5 citations) |
| remove hierarchical group knowledge transfer | CIFAR-100 · Top-1 accuracy | 84.38 -> 83.51 | Hurt | Boosting Residual Networks with Group Knowledge (AAAI 2024 · 5 citations) |
| Add Discriminative Area Enhancement (DAE) | ImageMeow, IPC10 · accuracy | 49.5 -> 52.6 | Helped | Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios (CVPR 2025 · 2 citations) |
| Combine DeiT feature distillation with MAE token-relation distillation | COCO · box AP | 47.5 -> 50.0 | Helped | Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners (ICLR 2024 · 2 citations) |
| Use GCMAE-PT instead of GCL-PT, GMAE-PT, or their sequential combination | HBN MDD LD · AUROC/ACC | 75.6/78.1 -> 77.6/79.4 | Helped | Pre-Training Graph Contrastive Masked Autoencoders are Strong Distillers for EEG (ICML 2025 · 1 citation) |
| Reintroduce the MLM objective during fine-tuning | CoLA · Matthew's Correlation Coefficient | 52.1 ± 4.0 -> 55.0 ± 3.0 | Helped | Induced Model Matching: Restricted Models Help Train Full-Featured Models (NeurIPS 2024 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 11 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-100 · 2 of 11 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 43 papers; from OpenAlex and the papers' affiliations).
- National University of Singapore · 4 papers
- Shanghai Jiao Tong University · 3 papers
- Google · 2 papers
- Independent Researcher · 2 papers
- University of Electronic Science and Technology of China · 2 papers
- Zhejiang University · 2 papers
- Google DeepMind · 2 papers
- Nanyang Technological University · 2 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Self and data-free distillation (57 papers)
- Distillation for machine translation (18 papers)
Elsewhere in the topic
- Rethinking distillation objectives (160 papers)
- Distillation with causal and adaptive guidance (82 papers)
- Few-shot and metric distillation (80 papers)
- Teacher properties and robustness (54 papers)
- Knowledge distillation for detectors (25 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).