Research map / Label-efficient and robust learning / Knowledge distillation
Teacher properties and robustness
Studies what makes teachers effective, including adversarial, random and noisy-input teachers.
54 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Knowledge distillation, keyword cluster “teacher student · kd · teachers”
State History Tried Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
10 papers in 2023–24 and 16 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 7.0% to 7.5%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · A Deep Hierarchical Approach to Lifelong Learning in Minecraft (AAAI 2017 · 221 citations)
- 2018 · Exercise-Enhanced Sequential Modeling for Student Performance Prediction (AAAI 2018 · 255 citations)
- 2019 · Knowledge Tracing Machines: Factorization Machines for Knowledge Tracing (AAAI 2019 · 178 citations)
- 2020 · Distilling Cross-Task Knowledge via Relationship Matching (CVPR 2020 · 28 citations)
- 2021 · Learning curves of generic features maps for realistic datasets with a teacher-student model (NeurIPS 2021 · 55 citations)
- 2022 · No Task Left Behind: Multi-Task Learning of Knowledge Tracing and Option Tracing for Better Student Assessment (AAAI 2022 · 17 citations)
- 2023 · PTADisc: A Cross-Course Dataset Supporting Personalized Learning in Cold-Start Scenarios (NeurIPS 2023 · 2 citations)
- 2024 · Boosting Neural Cognitive Diagnosis with Student’s Affective State Modeling (AAAI 2024 · 26 citations)
- 2025 · Weak-to-Strong Generalization Even in Random Feature Networks, Provably (ICML 2025 · 0 citations)
- 2026 · Debiased Cognitive Diagnosis: A Contrastive Counterfactual Modeling Method via Variational Autoencoder (AAAI 2026 · 1 citation)
Earliest papers here
- A Deep Hierarchical Approach to Lifelong Learning in Minecraft (AAAI 2017 · 221 citations)
- Progressive Prediction of Student Performance in College Programs (AAAI 2017 · 29 citations)
- Exercise-Enhanced Sequential Modeling for Student Performance Prediction (AAAI 2018 · 255 citations)
What has been tried
44 papers here report 415 measured changes (a component added, removed or swapped, compared with a baseline): helped 64%, no clear effect 10%, hurt 23%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| use only student affect traits for affect perception, rather than both student traits and exercise difficulty | ASSIST17 · ACC | 72.42 -> 63.54 | Hurt | Boosting Neural Cognitive Diagnosis with Student’s Affective State Modeling (AAAI 2024 · 26 citations) |
| add collaborative ability estimation | NIPS-EDU, 5 steps · intra-class ranking consistency | 0.7320 -> 0.7533 | Helped | Computerized Adaptive Testing via Collaborative Ranking (NeurIPS 2024 · 3 citations) |
| add distribution alignment with alpha=0.01 instead of alpha=0 | Stanford Cars · accuracy (All) | 63.1 -> 52.9 | Hurt | Flipped Classroom: Aligning Teacher Attention with Student in Generalized Category Discovery (NeurIPS 2024 · 3 citations) |
| Replace local data influence with LLM-as-a-Judge scoring | Alpaca Eval 2.0 · WR | 58.59% -> 54.93% | Hurt | Montessori-Instruct: Generate Influential Training Data Tailored for Student Learning (ICLR 2025 · 0 citations) |
| add the KL-Divergence gradient to the policy update | Vision-based manipulation · success rate | 0.47 ± 0.37 -> 0.88 ± 0.07 | Helped | Student-Informed Teacher Training (ICLR 2025 · 0 citations) |
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 26 papers; from OpenAlex and the papers' affiliations).
- University of Science and Technology of China · 2 papers
- University of California, Berkeley · 2 papers
- Princeton University · 2 papers
- Zhejiang University · 2 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Rethinking distillation objectives (160 papers)
- Distillation with causal and adaptive guidance (82 papers)
- Few-shot and metric distillation (80 papers)
- Knowledge distillation for detectors (25 papers)
Elsewhere in the topic
- Patient consistent distillation (68 papers)
- Self and data-free distillation (57 papers)
- Distillation for machine translation (18 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).