Research map / Label-efficient and robust learning / Knowledge distillation
Progressive distillation induces an implicit curriculum
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that progressive distillation accelerates student learning via an implicit curriculum in intermediate teacher checkpoints, with provable sample-complexity gains on sparse parity and empirical gains on PCFG and Wikipedia/Books BERT training.
Problem
Why progressive distillation (student learns from successive intermediate teacher checkpoints) helps when a better teacher does not always yield a better student; prior explanations focused on generalization, while the optimization benefit was unclear.
Approach
Intermediate teacher checkpoints taken during the phase transition provide an implicit curriculum (low-degree monomial correlations for sparse parity; shorter n-gram context dependence for PCFGs and natural language) that is absent from the converged teacher, accelerating optimization and reducing sample complexity.
Key results
- On (100,6)-sparse parity, a (2,1M) progressive distillation with the mid-phase-transition checkpoint reaches 100% accuracy, while other checkpoints do not help.
- Theory: progressive distillation with 2 checkpoints needs Θ̃(2^k d^2 ε^-2 + k^3) samples versus Ω(d^(k-1)) for one-shot distillation.
- On PCFG, the best single intermediate checkpoint is the inflection point C1, and progressive distillation improves student accuracy and feature measures over one-shot and cross-entropy training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a phase-transition intermediate teacher checkpoint in progressive distillation instead of one-shot distillation | Width-100 student on (100, 6)-sparse parity · (100, 6)-sparse parity | accuracy: 100% | Helped |
| Use progressive distillation with 2 checkpoints instead of one-shot distillation | Student learning (d, k)-sparse parity · (d, k)-sparse parity | sample complexity: Tilde Theta(2^k d^2 epsilon^-2 + k^3) vs. Omega(d^(k-1), epsilon^-2) | Helped |
| Use progressive distillation rather than one-shot or no distillation | 8-head BERT student on PCFG cfg3b · PCFG cfg3b masked prediction | top-1 accuracy | Helped |
| Use progressive distillation rather than one-shot or no distillation | 8-head BERT student on PCFG cfg3b · PCFG cfg3b | non-terminal prediction accuracy | Helped |
| Use progressive distillation rather than one-shot or no distillation | 4-head BERT student · Wikipedia and Books masked prediction | top-1 accuracy | Helped |
Surprising findings
- A properly chosen intermediate checkpoint with worse accuracy than the final teacher can yield a better student than the final teacher.
- Only the checkpoint during the phase transition helps; checkpoints just before or after it do not accelerate training.
Limitations
- Theory applies to large-batch SGD with small gradient noise, while experiments use online SGD with batch size 1.
- Temperature choices limited to default and very low values; effect of temperature not fully characterized.
- Generation-based (sampled) distillation is not studied; supervision is logit-based.
Experimental setup
Datasets: sparse parity synthetic data PCFG synthetic data (cfg3b, cfg3f) Wikipedia Books
Benchmarks: sparse parity accuracy PCFG masked prediction accuracy Wikipedia and Books masked prediction accuracy
Metrics: top-1 accuracy sample complexity total variation robustness measure non-terminal probe accuracy
Baselines: one-shot distillation cross-entropy training from data
Builds on: knowledge distillation BERT progressive distillation statistical query learning of sparse parity
Closest papers by meaning
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.89)
- Strong Teacher Not Needed? On Distillation in LLM Pretraining (NeurIPS 2026 · similarity 0.89)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.88)
- Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling (ICLR 2026 · similarity 0.87)
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.87)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.