Research map / Label-efficient and robust learning / Knowledge distillation
Distillation Scaling Laws
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a distillation scaling law predicting student cross-entropy from student size, distillation tokens, and teacher cross-entropy, and uses it to derive compute-optimal distillation recipes versus supervised learning.
Problem
Distillation pretraining consumes large compute, but it is unclear how to allocate compute between teacher and student, and when distillation beats supervised pretraining under equal resources.
Approach
A broken power law in teacher cross-entropy (capturing the capacity gap as a ratio of student and teacher learning capacities) combined with power laws in student size and distillation tokens predicts student loss; the teacher's size and tokens affect the student only through teacher cross-entropy.
Key results
- Supervised and distillation scaling laws fit observations at roughly ≲1% relative prediction error, including extrapolation from weaker to stronger models.
- Distillation outperforms supervised learning only up to a student-size-dependent compute threshold and only if a teacher already exists or is reused across students.
- When teacher pretraining is counted in compute, the best distilled student cross-entropy is always higher than the supervised student at the same total compute.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the distillation scaling law to predict student cross-entropy | Distillation scaling law · C4 English subset | relative prediction error: ≲1% | Helped |
| Distill rather than train with supervision when student compute or tokens are below the student-size-dependent threshold and the teacher cost is amortized | Student language models · C4 English subset | student cross-entropy | Helped |
| Use supervised learning rather than distillation when enough student compute or tokens are available | Student language models · C4 English subset | student cross-entropy | Hurt |
| Include teacher pretraining cost when training a teacher for a single student | Distillation with teacher pretraining and inference costs · C4 English subset | student cross-entropy | Hurt |
| Increase teacher capability beyond the student’s capacity | Distillation · C4 English subset | student cross-entropy | Hurt |
| Increase teacher size while it remains below student size | Distillation · C4 English subset | student error | Helped |
| Increase teacher size beyond student size | Distillation · C4 English subset | student error | Hurt |
| Set λ=1 instead of using the optimal λ* (ablation) | Distillation · C4 English subset | student cross-entropy | No clear effect |
| Set distillation temperature to τ=1 (ablation) | Distillation · C4 English subset | student performance | Helped |
Surprising findings
- When teacher training cost is included, distillation never beats supervised learning for a single student, contrary to the expectation that end-to-end distillation could be more efficient.
- Optimal temperature found to be 1, contrary to prior literature, which the authors attribute to their non-repeated data setup.
Limitations
- Only English C4 data with repeated data for large token runs, and only the same data distribution for teacher and student.
- Only transformer architectures studied and only pure distillation (lambda=1) with full-distribution KL; sequence-level distillation not studied.
- Scaling law coefficients are specific to the setup and not derived from first principles.
Experimental setup
Datasets: C4 (English subset)
Metrics: cross-entropy (validation loss) relative prediction error of scaling law fits calibration
Baselines: supervised learning (supervised scaling law)
Builds on: Chinchilla compute-optimal scaling laws knowledge distillation (Hinton et al.) muP SLSQP
Closest papers by meaning
- Strong Teacher Not Needed? On Distillation in LLM Pretraining (NeurIPS 2026 · similarity 0.88)
- SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision (NeurIPS 2026 · similarity 0.88)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.88)
- UniADS: Universal Architecture-Distiller Search for Distillation Gap (AAAI 2024 · similarity 0.87)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.87)
- Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.