Research map / Label-efficient and robust learning / Noisy labels

From Linear to Nonlinear: Provable Weak-to-Strong Generalization through Feature Learning

NeurIPS 2025 · Label-efficient and robust learning › Noisy labels · approach: Loss correction and noise robustness theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Provides a theoretical analysis of weak-to-strong generalization from a linear CNN teacher to a two-layer ReLU CNN student trained by gradient descent on patch-structured signal-plus-noise data, identifying data-scarce (benign vs harmful overfitting) and data-abundant (early-stopping-dependent) regimes.

Problem

Existing theory of weak-to-strong generalization is largely abstract or limited to linear/random-feature models, and does not show when gradient-based training of a nonlinear strong model under weak supervision generalizes.

Approach

Analyzes gradient descent on a two-layer ReLU CNN trained on pseudo-labels from a pretrained linear CNN over patch-structured data with easy signals, hard signals, and noise, deriving sample-size thresholds separating benign from harmful overfitting and showing early-stopped label correction in the data-abundant regime.</easy-core_idea>

Key results

• Weak linear CNN reaches 0.851 test accuracy on the synthetic data, with error bounded below by about p_h/2 due to hard-only data.
• On synthetic data with n_st=2000 the strong model's test accuracy exceeds the weak model's (benign overfitting), while n_st=75 stays near the weak model (harmful overfitting).
• With n_st=20000 test accuracy reaches near-perfect while overtraining degrades it; on MNIST the weak-to-strong gain rises then falls with n_st.

Measured effects reported in the paper

Surprising findings

• In the data-abundant regime, overtraining until zero training loss degrades test accuracy toward the weak model's level, even though good solutions exist.
• Strong model perfectly fitting flipped-label pseudo-labels can yield either low or high test error depending on data size, so early stopping is not strictly required in the finite-sample data-scarce regime.

Limitations

• Analysis restricted to simplified patch-structured data distribution with three patches and two-layer CNN with fixed second layer.
• Late-phase degradation in the data-abundant regime is observed empirically but not proven.
• Asymptotic regime with many polynomial conditions on parameters.

Experimental setup

Datasets: synthetic patch-structured data MNIST (modified)

Benchmarks: synthetic patch data modified MNIST

Metrics: test accuracy training accuracy weak-to-strong gain

Baselines: linear CNN weak model

Builds on: weak-to-strong generalization signal-noise decomposition benign overfitting

Closest papers by meaning

• Weak-to-Strong Generalization Even in Random Feature Networks, Provably (ICML 2025 · similarity 0.92)
• The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledge (NeurIPS 2026 · similarity 0.91)
• On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective (NeurIPS 2025 · similarity 0.91)
• Weak-to-Strong Generalization under Distribution Shifts (NeurIPS 2025 · similarity 0.90)
• Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss (ICML 2025 · similarity 0.90)
• Provable weak-to-strong generalization via benign overfitting (ICLR 2025 · similarity 0.89)