Research map / Label-efficient and robust learning / Knowledge distillation

Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimension

ICML 2025 · Label-efficient and robust learning › Knowledge distillation · approach: Teacher properties and robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes weak-to-strong (W2S) finetuning in ridgeless regression via intrinsic dimension and student-teacher feature correlation, showing W2S gains come from variance reduction in the discrepancy subspace between weak and strong features.

Problem

Explaining why a strong student finetuned on pseudo-labels from a weak teacher can outperform the weak teacher even when both have sufficient capacity for the task; what governs W2S generalization beyond absolute model capacity.

Approach

Casts W2S finetuning as ridgeless regression over low-intrinsic-dimension subspaces of strong and weak features, decomposing error into bias and variance. Shows the student inherits the teacher's variance in the shared subspace but reduces it by d_s/N in the discrepancy subspace, so feature discrepancy helps W2S.

Key results

• On synthetic regression with variance-dominated regime and low correlation dimension (d_{s∧w}=10), the W2S model outperforms both the weak teacher and the strong SFT baseline for moderate n and large N.
• On UTKFace, PGR and OPR scale inversely with labeled sample size n and show diminishing returns in pseudo-label size N; W2S performs better when d_s < d_w and when d_{s∧w}/d_w is lower.
• Injecting label noise into UTKFace improves relative W2S performance (PGR/OPR), taken as evidence that variance reduction is the key W2S advantage.

Measured effects reported in the paper

Surprising findings

• Larger labeled sample size n can make W2S degrade relative to the strong SFT baseline, even though absolute error keeps decreasing.
• Relative W2S performance can fail to beat the weak teacher when variance is not dominant (sigma^2 close to rho_s + rho_w).

Limitations

• Analysis is in the ridgeless regression / kernel regime with idealized exactly low-rank covariances; general case handled only with ridge and small-ball conditions.
• Advantage of W2S in bias is not quantified and is left to future work.
• Theory assumes Gaussian or sub-gaussian features and a linear probing view of finetuning.

Experimental setup

Datasets: UTKFace synthetic Gaussian regression

Benchmarks: UTKFace synthetic regression

Metrics: mean squared error excess risk performance gap recovery outperforming ratio

Baselines: strong SFT baseline strong ceiling model weak teacher

Builds on: weak-to-strong generalization intrinsic dimension of finetuning performance gap recovery

Closest papers by meaning

• On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective (NeurIPS 2025 · similarity 0.92)
• Weak-to-Strong Generalization Even in Random Feature Networks, Provably (ICML 2025 · similarity 0.91)
• Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions (ICML 2025 · similarity 0.89)
• Theoretical Analysis of Weak-to-Strong Generalization (NeurIPS 2024 · similarity 0.89)
• Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss (ICML 2025 · similarity 0.89)
• From Linear to Nonlinear: Provable Weak-to-Strong Generalization through Feature Learning (NeurIPS 2025 · similarity 0.89)