Research map / Learning theory and algorithms / Kernels and regression theory
Noisy Interpolation Learning with Shallow Univariate ReLU Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that minimum-norm interpolating two-layer univariate ReLU networks overfit in a tempered way for L_p losses with p<2 and catastrophically for p>=2, with grid-spaced samples giving tempered overfitting for all p.
Problem
Understanding whether interpolating noisy univariate data with minimum-norm two-layer ReLU networks yields tempered, benign, or catastrophic overfitting in terms of population L_p risk.
Approach
Provides a rigorous asymptotic analysis of min-norm univariate two-layer ReLU interpolators, characterizing spikes via curvature sign changes of the linear spline; tempered for p<2 (in probability, but infinite in expectation) and catastrophic for p>=2, with the catastrophe attributed to random spacing of i.i.d. inputs.
Key results
- For 1<=p<2, L_p risk of the min-norm interpolator is O(1/(2-p)) times L_p(f*) with high probability, and its expectation is infinite.
- For p>=2, L_p risk of the min-norm interpolator diverges to infinity in probability for Gaussian noise (catastrophic overfitting).
- With samples on a uniform grid, the min-norm interpolator is tempered for all p>=1, with L_p risk Theta(L_p(f*)).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use uniformly spaced training inputs instead of i.i.d. uniform inputs | minimum-norm ReLU interpolator · uniform-grid noisy regression | population L_p loss | Helped |
| use i.i.d. uniform training inputs instead of uniformly spaced inputs | minimum-norm ReLU interpolator · random-input noisy regression, p ≥ 2 | population L_p loss | Hurt |
| measure error with L_p for p ≥ 2 instead of L_p for 1 ≤ p < 2 | minimum-norm ReLU interpolator · i.i.d. noisy regression | population L_p loss | Hurt |
| measure error with L_p for p approaching 2 from below | minimum-norm ReLU interpolator · i.i.d. noisy regression | population L_p loss: 1/(2-p) | Hurt |
Surprising findings
- Tempered overfitting for p<2 holds in probability but the expected L_p risk is infinite, so the Mallinar et al. expectation-based definition would classify even p=1 as catastrophic.
- The square loss (p=2) is catastrophic, even though p slightly below 2 is tempered, with the tempered constant blowing up as p approaches 2.
Limitations
- Restricted to univariate inputs and uniform input distribution.
- Analyzes the min-norm interpolator with unpenalized skip connection rather than gradient-trained networks.
- No experiments reported in the provided text beyond illustrative figures.
Experimental setup
Metrics: L_p population risk convergence in probability expectation of L_p risk
Baselines: linear spline interpolator polynomial interpolation
Builds on: tempered overfitting (Mallinar et al. 2022) min-norm univariate ReLU network characterization (Savarese et al. 2019) linear spline interpolation
Closest papers by meaning
- Benign Overfitting in Adversarial Training of Neural Networks (ICML 2024 · similarity 0.91)
- Beyond Benign Overfitting in Nadaraya-Watson Interpolators (NeurIPS 2025 · similarity 0.89)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.89)
- Provable Tempered Overfitting of Minimal Nets and Typical Nets (NeurIPS 2024 · similarity 0.89)
- Rethinking Benign Overfitting in Two-Layer Neural Networks (ICML 2025 · similarity 0.89)
- Benign overfitting in leaky ReLU networks with moderate input dimension (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.