Research map / Label-efficient and robust learning / Recognition and long-tail
Simplicity Bias in Overparameterized Machine Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically argues that simplicity bias in overparameterized models arises from random construction (not uniform sampling) of functions, illustrated via random Boolean trees, wide-network Gaussian processes, and deep narrow ReLU network fixed-point convergence.
Problem
Lacking rigorous theoretical understanding of why overparameterized models (deep networks and others) generalize well despite interpolating training data; whether simplicity bias is universal and what its source is.
Approach
Simplicity bias comes from the fact that functions obtained by random construction (random trees, random weight initialization) are not uniformly sampled over the function space; wide networks are governed by the CLT (Gaussian process), while deep narrow networks are driven by contractive dynamical-system fixed points that map inputs to a constant.
Key results
- Theorem 1: with high probability, random Boolean trees interpolate the sample without overfitting when complexity L_f <= b·s/log n, for fixed epsilon and b with log(1/2+eps) < -b.
- Theorem 2: deep ReLU/ activation networks with i.i.d. zero-mean weights of std below 1/sqrt(w)·sigma'(0) have 0 as an asymptotically stable fixed point, so deep random networks output a near-constant function.
- Theorem 3: for narrow-deep networks under stated assumptions, out-of-sample predictions take training labels with probability increasing in cosine similarity to training inputs, i.e. nearest-neighbor/kernel-like behavior.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Construct a uniformly random Boolean tree and return one that fits the sample | Overparameterized Boolean function learning · Boolean function learning | out-of-sample agreement | Helped |
| Initialize a very wide network with random Gaussian weights | Wide neural networks · Noise-free data | simplicity bias | Helped |
| Train a wide network by gradient descent after random initialization | Wide neural networks · Noise-free data | generalization | Helped |
| Use a deep network with zero-mean random weights whose standard deviation is below the stated stability bound | Deep neural networks · Random initialization | convergence to a simple constant function | Helped |
| Increase the quasi-linear prologue depth before the network's first effective nonlinearity | Deep neural networks satisfying Theorem 3 · Out-of-sample prediction | probability of outputting a training label | Helped |
| Use the trained network under Theorem 3's assumptions | Deep neural networks satisfying Theorem 3 · Out-of-sample prediction | dependence of label probability on cosine similarity | Helped |
| Replace a wide-network mechanism with a deep, narrow network's dynamical-system mechanism | Neural networks · Simplicity bias | simplicity bias | Mixed |
| Use different optimizers while retaining the random-construction bias | Overparameterized learning models · Generalization | generalization | No clear effect |
Surprising findings
- Randomly constructed Boolean trees that perfectly fit a sample generalize at chance-free levels: the naive random construction algorithm avoids overfitting when f is simple enough relative to sample size.
Limitations
- Results are theoretical and rely on idealized assumptions (orthonormal samples, linear-near-origin activations, quasi-linear prologue, large l_b)
- The naive sampling algorithms are computationally inefficient and non-constructive; no experiments on practical training are reported
- Claim that the source is universal is conjectural beyond the concrete examples
Experimental setup
Builds on: Chauvin et al. 2004 uniform Boolean tree distribution Jacot et al. 2018 neural tangent kernel Xiao et al. 2020 deep information propagation Lee et al. 2019 wide networks as Gaussian processes
Closest papers by meaning
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.92)
- Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data (ICML 2024 · similarity 0.90)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.90)
- Position: Deep Learning is Not So Mysterious or Different (ICML 2025 · similarity 0.90)
- Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization (NeurIPS 2024 · similarity 0.89)
- The Rich and the Simple: On the Implicit Bias of Adam and SGD (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.