Research map / Optimization / Implicit bias and generalization
Towards Understanding Knowledge Distillation
Mary Phuong, Christoph Lampert
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distillation-trained linear student with n >= d training points (exact recovery of teacher weights) | linear distillation (N=1, gradient flow from zero init) · theoretical, linear binary classification | expected transfer risk | Improved |
| increase transfer set size from n < d to n >= d | linear distillation student · theoretical, linear binary classification | expected transfer risk | Improved |
| favorable data geometry (higher angular alignment, kappa) instead of low alignment (ablation) | linear distillation student · synthetic polynomial-angle tasks, d=1000, n=20 | transfer risk | Improved |
| gradient-descent optimization bias (small δ) instead of treating all minimizers equally (large δ) (ablation) | linear distillation student · MNIST digits 0 vs 1, n=100 | transfer risk | Improved |
| higher monotonicity index of the learner (negative correlation with transfer risk) (ablation) | distillation and hard-target learners · polynomial-angle task, d=100, n=5 | transfer risk | No clear effect |
Closest papers by meaning
- Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher (NeurIPS 2020 · similarity 0.90)
- Knowledge Distillation: Bad Models Can Be Good Role Models (NeurIPS 2022 · similarity 0.89)
- SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelines (ICLR 2026 · similarity 0.89)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.88)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.87)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
