Research map / Optimization / Implicit bias and generalization
Benign overfitting in leaky ReLU networks with moderate input dimension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that two-layer leaky ReLU networks trained by gradient descent on the hinge loss exhibit benign or harmful overfitting depending on SNR, using approximate margin maximization and requiring only d = Omega(n) rather than d = Omega(n^2 log n).
Problem
Characterizing when benign versus harmful overfitting occurs for shallow leaky ReLU networks trained on noisy binary classification data, previously requiring nearly orthogonal inputs and d = Omega(n^2 log n).
Approach
Introduces approximate margin maximization (network norm bounded by a constant times the max-margin linear classifier norm) and shows GD on hinge loss for leaky ReLU networks satisfies it; uses well-conditioned Gaussian noise matrices (singular value bounds) instead of near-orthogonality to derive benign and non-benign overfitting conditions on SNR.
Key results
- Benign overfitting holds under d = Omega(n), gamma = Omega(1/k), with test error bounded by exp(-C d/(k(1+m|A|^2)))
- Non-benign overfitting: if d = Omega(n) and gamma = O(alpha^3/d), the generalization error is at least 1/8
- GD on hinge loss for leaky ReLU networks converges in finite time with approximate margin factor |A_GD| <= C/(alpha sqrt(m))
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use signal strength γ = Ω(1/k) with d = Ω(n) and k = O(α²n) | Leaky ReLU + GD + hinge loss · Binary classification with flipped training labels | generalization error: \leq \exp\left(-C\cdot\frac{\alpha^2 d}{k}\right) | Helped |
| Use low signal strength γ = O(α³/d) | Leaky ReLU + GD + hinge loss · Binary classification with flipped training labels | generalization error: \geq \frac{1}{8} | Hurt |
| Train with gradient descent on hinge loss under the stated step-size and initialization assumptions | Leaky ReLU network on linearly separable data · Linearly separable training data | training loss: L(\mW, \mX, \hat{\vy}) = 0 | Helped |
| Require d = Ω(n) rather than the prior-work requirement d = Ω(n² log n) | Benign-overfitting analysis for leaky ReLU networks · Noisy binary classification | input dimension requirement: d = Ω(n) ; d = Ω(n² log n) | Helped |
Limitations
- Requires the training data to be linearly separable
- Restricted to two-layer networks with fixed outer-layer weights
- Analysis limited to hinge loss and a Gaussian cluster data model
Experimental setup
Datasets: synthetic Gaussian mixture data model
Metrics: generalization error bound misclassification probability
Baselines: Frei et al. benign overfitting in leaky ReLU networks Kou et al. ReLU networks with exponential losses Cao et al. convolutional benign overfitting
Builds on: implicit bias of gradient descent (homogeneous networks) perceptron convergence proof
Closest papers by meaning
- Rethinking Benign Overfitting in Two-Layer Neural Networks (ICML 2025 · similarity 0.90)
- Benign Overfitting in Adversarial Training of Neural Networks (ICML 2024 · similarity 0.89)
- Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data (ICLR 2024 · similarity 0.89)
- Noisy Interpolation Learning with Shallow Univariate ReLU Networks (ICLR 2024 · similarity 0.88)
- Benign Overfitting in Single-Head Attention (NeurIPS 2025 · similarity 0.88)
- Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.