Research map / Label-efficient and robust learning / Knowledge distillation
High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a sharp non-asymptotic risk characterization of knowledge distillation / surrogate-to-target training for ridgeless high-dimensional linear regression, derives the optimal surrogate and feature-masking rule, and shows weak-to-strong gains that do not change the scaling exponent.
Problem
Understanding the excess risk, optimal surrogate (teacher/weak model) and the effect of sample sizes and distribution shift in knowledge distillation and weak-to-strong training, where a target model is fit on labels from a surrogate model.
Approach
Uses the non-asymptotic Gaussian min-max characterization of the minimum-norm interpolator to derive the exact risk of the surrogate-to-target pipeline; shows the optimal surrogate amplifies principal features and shrinks tail features (with a threshold set by covariance statistics), and that optionally masking weak features beats ground-truth labels, while the scaling exponent is unchanged.
Key results
- Optimal surrogate amplifies principal coefficients and shrinks tail coefficients; masking features beyond a threshold strictly beats ground-truth labels when covariance is not isotropic.
- Under power-law spectra the optimal surrogate-to-target risk scales as Theta(n^{-(beta-1)}) or Theta(n^{-2alpha}), identical to the standard target model's scaling exponent.
- On CIFAR-10 with ResNet-50, surrogate-to-target models beat their surrogates but do not outperform the standard target model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the risk-minimizing surrogate parameter instead of ground-truth labels | Surrogate-to-target ridgeless regression · Over-parameterized regression (p > n) | asymptotic excess test risk | Helped |
| Select features satisfying 1 - ζ_i^2 > Ω and discard the rest | Weak-to-strong surrogate-to-target regression · Over-parameterized regression (p > n) | asymptotic excess test risk | Helped |
| Use the optimal surrogate parameter rather than ground-truth labels | Surrogate-to-target ridgeless regression · Power-law eigenvalues and signal coefficients | excess test risk scaling | No clear effect |
| Use a surrogate parameter different from ground truth | Surrogate-to-target regression · Under-parameterized region (n > p) | asymptotic excess risk | No clear effect |
| Train the target model on weak-model predictions rather than ground-truth labels | Pretrained ResNet-50 fine-tuned on CIFAR-10 · CIFAR-10 | classification performance | Hurt |
| Train the target model on weak-model predictions rather than use the weak model directly | Pretrained ResNet-50 fine-tuned on CIFAR-10 · CIFAR-10 | classification performance | Helped |
| Discard tail features beyond the pruning threshold | Optimal masked surrogate · Ridgeless regression with diagonal target covariance | asymptotic excess test risk | Helped |
Surprising findings
- Weak-to-strong training with an optimally chosen weak surrogate can strictly outperform training on true labels under the same data budget, yet it cannot improve the data scaling exponent.
- In the CIFAR-10 experiment, surrogate-to-target models underperform the standard strong target model.
Limitations
- Analysis restricted to linear / ridgeless regression with proportional asymptotics; neural networks left open.
- Optimal surrogate requires knowledge of the target covariance and ground-truth parameter, not practically available.
- Experiments with neural networks do not reproduce outperformance over the strong model.
Experimental setup
Datasets: CIFAR-10 synthetic Gaussian linear regression data
Benchmarks: CIFAR-10
Metrics: test risk (excess) scaling exponent
Baselines: standard target model with ground-truth labels covariance shift model
Builds on: minimum norm interpolator characterization (Gaussian min-max) covariate shift transfer learning analysis benign overfitting
Closest papers by meaning
- Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimension (ICML 2025 · similarity 0.88)
- Scaling laws for learning with real and surrogate data (NeurIPS 2024 · similarity 0.88)
- On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective (NeurIPS 2025 · similarity 0.87)
- From Linear to Nonlinear: Provable Weak-to-Strong Generalization through Feature Learning (NeurIPS 2025 · similarity 0.87)
- Weak-to-Strong Generalization Even in Random Feature Networks, Provably (ICML 2025 · similarity 0.87)
- Provable weak-to-strong generalization via benign overfitting (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.