Research map / Label-efficient and robust learning / Knowledge distillation
On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical analysis of weak-to-strong generalization in high-dimensional ridge and weighted ridge regression plus a nonlinear multi-index two-layer network, identifying three mechanisms: teacher under-regularization compensation, task-aligned student regularization, and easy-feature transfer combined with pretrained hard features.
Problem
Weak-to-strong generalization (a student trained on a weaker teacher's labels surpassing the teacher) is widely observed empirically, but its mechanisms are poorly understood, especially the roles of teacher/student regularization, parameterization regime, and feature learning.
Approach
Derives asymptotic test errors for teacher-student ridge and weighted-ridge regression in the high-dimensional proportional limit, showing the student can compensate for teacher under-regularization or use a task-aligned regularizer; in a nonlinear multi-index setting, a pretrained student learns hard features while inheriting easy features from the teacher after one gradient step.
Key results
- Theorem gives asymptotic student-teacher test error difference for ridge regression; if teacher is over-regularized (lambda_t >= sigma^2 gamma_t) the student can never outperform it, while under-regularized teachers can be outperformed by a student with suitable lambda_s when gamma_s<1.
- Weighted ridge student: test error difference becomes Delta - zeta^2 Delta_Gamma, so a student with better-aligned regularization can beat an optimally regularized teacher even when Delta >= 0.
- In the nonlinear multi-index setting, after one gradient step the student retains alignment to the hard direction (||W_s beta_h||_op -> c_h > 0) while learning the easy direction, whereas the teacher fails to learn the hard direction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adequately regularize a student trained from an under-regularized ridge teacher | Ridge student · High-dimensional ridge regression | test error | Helped |
| Use a ridge teacher whose regularization is at least the optimal value | Ridge student · High-dimensional ridge regression | test error | No clear effect |
| Use an under-parameterized student with suitable regularization | Ridge student · High-dimensional ridge regression, γ_s < 1 | test error | Helped |
| Use an over-parameterized student with regularization in the qualifying interval | Ridge student · High-dimensional ridge regression, γ_s > 1 | test error | Helped |
| Use an over-parameterized student when the stated parameter condition does not hold | Ridge student · High-dimensional ridge regression, γ_s > 1 | test error | Hurt |
| Replace standard ridge with task-aligned weighted ridge regularization | Weighted ridge student · High-dimensional weighted ridge regression | test error | Helped |
| Increase alignment of the student's learned direction with the target | Weighted ridge student · High-dimensional weighted ridge regression | test error | Helped |
| Give the student a task-aligned regularization structure | Weighted ridge student · High-dimensional weighted ridge regression with an over-regularized teacher | test error | Helped |
| Pretrain the student to align with the hard, transferable direction | Nonlinear student · Nonlinear multi-index setting | alignment with the hard direction | Helped |
| Use one student gradient step to learn the easy direction from teacher labels while preserving pretrained hard-direction alignment | Nonlinear student · Nonlinear multi-index setting | alignment with easy and hard directions | Helped |
| Set the teacher to its optimal ridge regularization | Ridge student · High-dimensional ridge regression, student test error versus γ_t | test error | Helped |
| Set the teacher to its optimal ridge regularization | Ridge student · High-dimensional ridge regression | test error relative to teacher | Hurt |
Surprising findings
- Optimal teacher regularization eliminates double descent in the student's test error, yet the student still cannot outperform the teacher in that case.
- In the weighted ridge setting, pairs with an over-regularized teacher (lambda_t > sigma^2 gamma_t) can still yield a student that outperforms the teacher.
Limitations
- Results are proven only in the high-dimensional proportional asymptotic limit with Gaussian data and linear or simplified two-layer models.
- Nonlinear analysis relies on a single gradient step and specific assumptions (kappa_e=1, kappa_h>1, spiked pretrained features), not full training dynamics.
- No experiments with real foundation models; validation is limited to synthetic simulations.
Experimental setup
Datasets: synthetic Gaussian linear regression data synthetic multi-index data
Metrics: test error (L_s, L_t) test error difference
Builds on: Marchenko-Pastur law weak recovery in two-layer networks information exponent
Closest papers by meaning
- Weak-to-Strong Generalization Even in Random Feature Networks, Provably (ICML 2025 · similarity 0.94)
- Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimension (ICML 2025 · similarity 0.92)
- Theoretical Analysis of Weak-to-Strong Generalization (NeurIPS 2024 · similarity 0.92)
- From Linear to Nonlinear: Provable Weak-to-Strong Generalization through Feature Learning (NeurIPS 2025 · similarity 0.91)
- Provable weak-to-strong generalization via benign overfitting (ICLR 2025 · similarity 0.91)
- Quantifying the Gain in Weak-to-Strong Generalization (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.