Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a higher soft-label ratio | Knowledge distillation with a perfect teacher · Wide-network distillation | data inefficiency | Helped |
| Early-stop the teacher | Knowledge distillation · Wide-network distillation | data inefficiency | Helped |
| Add hard labels to soft-label distillation | Knowledge distillation with a perfect teacher · Wide-network distillation | data inefficiency | Hurt |
| Use pure soft-label distillation rather than mixing in hard labels | Knowledge distillation with a perfect teacher · Wide-network distillation | transfer risk convergence with sample size | Helped |
| Add a small portion of hard labels | Knowledge distillation from an imperfect teacher · Wide-network distillation | student weight alignment with the ground-truth direction | Helped |
| Add hard labels when the teacher is sufficiently accurate | Knowledge distillation from an accurate teacher · Wide-network distillation | student weight alignment with the ground-truth direction | Hurt |
Closest papers by meaning
- Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation (NeurIPS 2022 · similarity 0.92)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.91)
- Improved Knowledge Distillation via Teacher Assistant (AAAI 2020 · similarity 0.91)
- Revisiting Knowledge Distillation via Label Smoothing Regularization (CVPR 2020 · similarity 0.90)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.89)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.