Research map / Label-efficient and robust learning / Knowledge distillation
Provable weak-to-strong generalization via benign overfitting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical analysis of weak-to-strong generalization in an overparameterized spiked covariance model with minimum-l2-norm interpolation, showing successful generalization vs random guessing phases determined by bi-level ensemble parameters.
Problem
Provably explaining when a strong student trained on pseudolabels from a weak teacher generalizes versus merely imitating the weak teacher's errors, in a stylized overparameterized linear setting.
Approach
Models weak-to-strong finetuning as minimum l2-norm interpolation on Gaussian features from a bi-level spiked covariance ensemble, decomposing the strong model's output into signal survival and contamination of the true direction, and deriving exact phase boundaries; also proves a tight lower tail bound for maxima of correlated Gaussians.
Key results
- Under the subset ensemble, the strong model generalizes (test error o(1)) when u > q_w + r_w - min{1-r, tau_s}, and otherwise approaches random guessing (error 1/2 - o(1)).
- Clean-label binary error for the strong model is 1/2 - (1/pi) arctan(Theta(n^{tau_s})), with tau_s = p+1-2(q+r).
- Simulations with n=50 confirm that weak accuracy is near random and weak-to-strong accuracy increases with m in the predicted regimes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the strong model on sufficiently many weak pseudolabels, with u > q_{\weak} + r_{\weak} - min{1-r, τ_{\strong}} | Weak-to-strong MNI · Binary classification, weak-to-strong subset ensemble | test error: \frac{1}{2} - o_n(1) -> o_n(1) | Helped |
| Train the strong model on weak pseudolabels below the stated sample threshold | Weak-to-strong MNI · Binary classification, weak-to-strong subset ensemble | test error: \frac{1}{2} - o_n(1) | No clear effect |
| Use weak features and clean labels for the weak model when τ_weak < 0 | Weak MNI · Binary classification, n clean labels | test error: \frac{1}{2} - o_n(1) | No clear effect |
| Use hard multilabel pseudolabels and train the strong model in the stated regimes | Weak-to-strong MNI · Multilabel classification | test error | Helped |
Surprising findings
- Weak-to-strong generalization can succeed even though the weak labels are asymptotically random guesses relative to the truth.
- In practice, transitions are not sharp (PGR below 1) because the asymptotic analysis does not capture finite-n rates and the 1-sparse assumption is idealized.
Limitations
- Relies on Gaussian features, a 1-sparse true label assumption, and a subset ensemble between weak and strong features.
- Multiclass weak-to-strong result is not fully proven; only multilabel and binary cases are established.
- Third-assumption on number of weak examples is technical and believed but not proven essentially tight.
Experimental setup
Metrics: test error (0-1 loss) test accuracy
Baselines: strong model trained on clean labels weak model PCA
Builds on: benign overfitting (bi-level ensemble) weak-to-strong generalization (Burns et al. 2023) Hanson-Wright inequality
Closest papers by meaning
- Weak-to-Strong Generalization Even in Random Feature Networks, Provably (ICML 2025 · similarity 0.91)
- Theoretical Analysis of Weak-to-Strong Generalization (NeurIPS 2024 · similarity 0.91)
- Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss (ICML 2025 · similarity 0.91)
- On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective (NeurIPS 2025 · similarity 0.91)
- From Linear to Nonlinear: Provable Weak-to-Strong Generalization through Feature Learning (NeurIPS 2025 · similarity 0.89)
- Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimension (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.