Research map / Optimization / Implicit bias and generalization

Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks

ICML 2025 · Optimization › Implicit bias and generalization · approach: Shallow ReLU network convergence · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows theoretically that small-initialized overparametrized two-layer ReLU networks trained by gradient flow, on linear data with many samples (n ≳ d^3 log d), converge to a non-interpolating spurious stationary point approximating OLS, via early alignment and an optimization threshold.

Problem

Overparametrized neural networks trained with gradient descent may fail to interpolate training data for large sample sizes; the paper asks why and whether this non-interpolation yields simplicity bias and better test loss.

Approach

Early alignment of neurons toward extremal vectors of a function G_n whose gradient concentrates to the true-loss gradient D(w) as n grows; for linear teachers with two extremal directions, the network converges to the difference of two ReLU neurons equal to the OLS estimators on positive and negative data halves, giving an optimization threshold beyond which interpolation does not occur.

Key results

• For d=5, interpolation stops around n≈3000 samples; beyond this threshold the learned estimator approximates OLS with test loss near the noise level.
• With n=500, 69% of neurons have |cosine| < 0.9 with OLS; with n=5000, 91% have |cosine| > 0.9.
• Theorem gives threshold n* = Θ(d^3 log d) and shows convergence to a non-interpolating stationary point equal to the difference of two ReLU neurons with OLS directions.

Measured effects reported in the paper

Surprising findings

• Training converges to a spurious non-interpolating stationary point rather than a global minimum, and this non-interpolation improves test loss.
• GeLU networks fail to interpolate even at n=500 in the reported experiments.

Limitations

• Theory restricted to a linear teacher, a single activation cone and a symmetric data distribution with strong assumptions on initialization.
• Threshold requires very large n relative to d (n ≳ d^3 log d); d=10 experiments show a larger threshold than predicted.
• Only synthetic toy experiments; the link to in-context learning and diffusion remains conjectural.

Experimental setup

Datasets: synthetic linear regression data

Benchmarks: synthetic linear regression

Metrics: train loss test loss cosine similarity with OLS estimator

Baselines: ordinary least squares interpolating global minimum

Builds on: early alignment analysis of small-initialized two-layer networks Clarke subdifferential gradient flow

Closest papers by meaning

• Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes (NeurIPS 2024 · similarity 0.91)
• Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.91)
• Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data (ICML 2024 · similarity 0.90)
• Simplicity Bias in Overparameterized Machine Learning (AAAI 2024 · similarity 0.90)
• Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.90)
• Noisy Interpolation Learning with Shallow Univariate ReLU Networks (ICLR 2024 · similarity 0.89)