Research map / Label-efficient and robust learning / Knowledge distillation
Understanding the Gains from Repeated Self-Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically, for fixed-design linear regression, that optimally tuned multi-step self-distillation (r steps, r = rank of X) can cut excess risk by up to a factor of d versus one-step SD, and validates with UCI regression experiments (up to 47% MSE reduction).
Problem
Characterizing how much gain repeated (multi-step) self-distillation can provide over one-step self-distillation and ridge, in particular for linear regression.
Approach
Model k-step self-distillation with ridge teacher/student and per-step imitation parameters as a preconditioned ridge estimator, showing the excess risk is quadratic in reparametrized imitation weights and that r steps achieve a fundamental lower bound for linear estimators under singular-value and alignment assumptions.
Key results
- Theory: optimally tuned r-step SD can reduce excess risk by a factor up to r (= d in full-rank case) relative to 1-step SD and ridge.
- On Air Quality UCI data, optimal 2-step SD reduces test MSE from 2.01 (ridge) to 1.06, a 47.2% reduction.
- On AEP, SD does not improve over ridge (MSE 0.62 vs 0.63 for 2-step SD).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 2-step SD instead of ridge | Linear regression · Air Quality test set | MSE: 2.01 -> 1.06 | Helped |
| Use 2-step SD instead of 1-step SD | Linear regression · Air Quality test set | MSE: 1.99 -> 1.06 | Helped |
| Use 2-step SD instead of ridge | Linear regression · Airfoil test set | MSE: 1.34 -> 1.19 | Helped |
| Use 2-step SD instead of 1-step SD | Linear regression · Airfoil test set | MSE: 1.22 -> 1.19 | Helped |
| Use 1-step SD instead of ridge | Linear regression · AEP test set | MSE: 0.62 -> 0.62 | No clear effect |
| Use 2-step SD instead of ridge | Linear regression · AEP test set | MSE: 0.62 -> 0.63 | Hurt |
| Use r-step SD instead of 1-step SD | Fixed-design linear regression under the theorem's assumptions · Theorem problem family | excess risk: \left( \nicefrac{0.99}{2^{9}} \right) \, \frac{r\gamma^2}{n} -> \frac{\gamma^2}{n} | Helped |
| Increase the number of optimally tuned self-distillation steps (ablation) | Repeated self-distillation · Fixed-design synthetic linear regression | excess risk | Helped |
| Use repeated SD when all non-zero singular values are equal (ablation) | Ridge estimator · Fixed-design linear regression with equal singular values | excess risk | No clear effect |
| Use repeated SD when the signal is equally aligned with all eigenvectors (ablation) | Ridge estimator · Fixed-design linear regression with equal signal components | excess risk | No clear effect |
Surprising findings
- The multiplicative separation between r-step and 1-step SD can be as large as Omega(r), which is unexpected given prior single-step analyses.
- On the AEP dataset, multi-step SD gives no improvement over optimal ridge.
Limitations
- Analysis restricted to fixed-design linear regression; random design left for future work.
- Separation requires assumptions on distinct singular values and alignment of theta* with eigenvectors; violating them removes the gain.
- Requires extra hyperparameter tuning of imitation parameters, and multi-step training costs more compute.
Experimental setup
Datasets: Air Quality (UCI) Airfoil (UCI) AEP (UCI)
Benchmarks: Air Quality Airfoil AEP
Metrics: mean squared error excess risk
Baselines: optimal ridge 1-step self-distillation
Builds on: Born-again networks 1-step self-distillation theory (Mou et al. / Dong et al.) ridge regression
Closest papers by meaning
- Self-Distillation Amplifies Regularization in Hilbert Space (NeurIPS 2020 · similarity 0.86)
- Distillation Scaling Laws (ICML 2025 · similarity 0.86)
- Strong Teacher Not Needed? On Distillation in LLM Pretraining (NeurIPS 2026 · similarity 0.84)
- The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model (NeurIPS 2025 · similarity 0.83)
- Knowledge Distillation with Auxiliary Variable (ICML 2024 · similarity 0.83)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.