Research map / Label-efficient and robust learning / Knowledge distillation
Weak-to-Strong Generalization Even in Random Feature Networks, Provably
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows weak-to-strong generalization occurs in random feature networks (ReLU and linear) with a larger early-stopped student, achieving quadratic error reduction, and proves a general lower bound showing the student error is at least order TL^2 for shrinkage-optimal learners.
Problem
Whether a strong student can outperform a weak teacher purely due to size advantage in simple non-pretrained models, and what fundamental limits exist on such weak-to-strong generalization.
Approach
Analyzes teacher and student as random feature two-layer networks trained via early-stopped gradient flow; shows the student's error can be quadratically (or ~1.49 power for ReLU under Gaussian universality) smaller than the teacher's, and proves that any shrinkage-optimal student and teacher cannot go below ~(3/4)TL^2, so weak-to-strong gains are bounded.
Key results
- Linear random feature networks: student loss SL = Õ(k TL^2) with high probability, so PGR approaches 1.
- General lower bound: SL >= (sqrt(1+3TL)-sqrt(1-TL))^2/4 >= (3/4) TL^2 for shrinkage-optimal teacher and student, so quadratic gain is the maximum.
- ReLU random feature networks under Gaussian universality: SL = Õ(TL^1.49).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase student feature count relative to the teacher and train on teacher-generated labels | Random feature networks · ReLU random feature networks | population MSE: L_S≤O_{d,k}((log^2 M_T+log^2 L_T)/√M_T) L_T | Helped |
| Use a larger student with linear random features and anisotropic inputs | Linear random feature networks · Linear random feature networks | population MSE: L_S≤\tilde{O}(kL_T^2) | Helped |
| Apply early stopping when training the student on teacher labels | Random feature student · ReLU random feature networks | student-to-teacher loss ratio | Helped |
| Continue student training past the useful stopping time | Random feature student · ReLU random feature networks | student-to-teacher loss ratio | Hurt |
| Train the student to convergence instead of stopping early | Random feature student · ReLU and linear random feature networks | population MSE | Hurt |
| Increase teacher size while selecting the appropriate student stopping time | ReLU random feature student · Linear target on the sphere | student-to-teacher loss ratio | Helped |
| Increase teacher size while selecting the appropriate student stopping time | Linear random feature student · Anisotropic linear target | student-to-squared-teacher-loss ratio: L_S/L_T^2 converges to approximately 1 | Helped |
| Use early-stopped gradient flow from zero initialization rather than unrestricted teacher imitation | Random feature student · Random feature weak-to-strong training | population MSE: L_S≥((√(1+3L_T)-√(1-L_T))^2)/4≥3/4 L_T^2 | Helped |
| Bootstrap through a chain of students, each trained on the previous model | Weak-to-strong student chain · Bootstrapped weak-to-strong learning | population MSE: L_ST,i≥(1-√(1-L_T))^2≥1/4 L_T^2 | No clear effect |
| Use a student with features that are more specialized or pre-trained, but train its top layer from zero | Weak-to-strong student · General shrinkage-optimal learning | population MSE: L_S≥((√(1+3L_T)-√(1-L_T))^2)/4≥3/4 L_T^2 | No clear effect |
| Fine-tune a pre-trained top layer from its pre-trained initialization rather than from the origin | Pre-trained student · General weak-to-strong learning | population MSE | Mixed |
Surprising findings
- Weak-to-strong gains are fundamentally bounded: a student cannot boost a slightly-better-than-chance teacher to low error.
- Pre-trained student features do not escape the limit if the top layer is trained from zero initialization.
Limitations
- Lower bound applies only to gradient flow initialized at origin or convex classes containing origin; fine-tuning from pretrained init escapes it.
- ReLU polynomial exponent result relies on the Gaussian Universality Ansatz.
- Analysis uses infinite-width student limit.
Experimental setup
Metrics: student loss vs teacher loss ratio performance gap recovered
Builds on: Burns et al. 2024 weak-to-strong generalization random feature models neural tangent kernel
Closest papers by meaning
- On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective (NeurIPS 2025 · similarity 0.94)
- From Linear to Nonlinear: Provable Weak-to-Strong Generalization through Feature Learning (NeurIPS 2025 · similarity 0.92)
- Theoretical Analysis of Weak-to-Strong Generalization (NeurIPS 2024 · similarity 0.92)
- Provable weak-to-strong generalization via benign overfitting (ICLR 2025 · similarity 0.91)
- Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimension (ICML 2025 · similarity 0.91)
- Quantifying the Gain in Weak-to-Strong Generalization (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.