Research map / Optimization / Implicit bias and generalization
How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that uniformly sampled quantized or continuous neural nets conditioned on interpolating training data generalize well when a narrow teacher network generates the labels, with sample complexity proportional to teacher size times log-quantization bits.
Problem
Why over-parameterized neural networks generalize when randomly sampled from a seemingly uniform parameter prior and conditioned on zero training error (random interpolators), as observed empirically.
Approach
Parameter redundancy in NNs (zeroed neurons, zero scales) makes a uniform prior over parameters induce a non-uniform prior over functions favoring narrow teachers; posterior (guess-and-check) samples thus have sample complexity roughly the teacher's complexity rather than the student's.
Key results
- For quantized fully connected nets, the effective sample complexity is bounded by sum_l (d*_l d*_{l-1} + 2 d_l) log Q for scaled-neuron nets, approximately the teacher's bit count plus a linear term in student width.
- Posterior (guess-and-check) samples with N = (C~ + 3 log(2/delta))/eps achieve population error at most eps with probability 1-delta.
- A corollary shows that the volume of high-error interpolators is exponentially small in the training set size with high probability over the data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add per-neuron scaling parameters to the quantized FC student | Quantized fully connected networks · Quantized fully connected networks with a narrow teacher | effective sample complexity upper bound: \BSCM{FC} \triangleq \rb{\sum_{l=1}^{L}{\rb{d_{l}^{\star}d_{l-1}+d_{l}^{\star}}}}\log Q -> \BSCM{SFC} \triangleq \rb{\su | Helped |
| Add per-channel scaling parameters to the quantized convolutional student | Quantized convolutional networks · Quantized convolutional networks with a narrow teacher | effective sample complexity upper bound: \BSCM{CNN}=&\left(d_s + 1 + \sum_{l=1}^L \rb{k_l c_{l}^{\star}c_{l-1} + c_{l}^{\star}} \right) \log Q -> \BSCM{SCNN}=& \ | Helped |
| Use a narrow teacher with a uniform prior over student parameters | Posterior sampling (Guess and Check) · Realizable binary classification | sample complexity for generalization: O\left(-\log \tilde{p}\right) | Helped |
| Use a continuous spherical prior with angular-margin assumptions instead of a quantized prior | Continuous two-layer FC networks · Continuous two-layer networks | effective sample complexity bound | Mixed |
Surprising findings
- Sampling a uniform random network conditioned on interpolation yields a bias toward narrow teachers without any explicit sparsity or width regularization.
Limitations
- Requires existence of a narrow teacher that perfectly labels data, which may not hold for real datasets.
- Bounds for continuous nets rely on angular margin assumptions that may decay with data size and dimension.
- Bounds are not validated empirically; teacher existence for ImageNet is not directly verified.
Experimental setup
Builds on: PAC-Bayes Guess and Check (Valle-Perez et al.) Minimum description length
Closest papers by meaning
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.89)
- Neural Redshift: Random Networks are not Random Functions (CVPR 2024 · similarity 0.89)
- How much does Initialization Affect Generalization? (ICML 2023 · similarity 0.89)
- Simplicity Bias in Overparameterized Machine Learning (AAAI 2024 · similarity 0.89)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.89)
- Noisy Interpolation Learning with Shallow Univariate ReLU Networks (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.