Research map / Label-efficient and robust learning / Noisy labels
Rethinking Benign Overfitting in Two-Layer Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Refines the feature-noise data model with class-dependent noise covariances and proves a benign-vs-harmful overfitting phase transition for two-layer ReLU CNNs, showing memorized class-dependent noise can help long-tailed generalization.
Problem
Prior feature-noise theory predicts harmful overfitting when noise is large, yet overparameterized networks rarely overfit harmfully and empirically memorizing atypical long-tailed examples helps generalization; the paper asks how to theoretically explain benign overfitting in such networks.
Approach
Replace the homogeneous noise model with class-dependent noise covariances A_y, so neurons learn implicit features from class-specific noise; derive test-loss bounds in terms of signal-to-noise ratio and a noise correlation ratio, and a training-free influence score based on intra-class covariance Frobenius norm.
Key results
- Theory: under a large noise correlation ratio Gamma, long-tailed class test loss is bounded by sum_j exp(-c_2 L^2 Gamma_{k,j}), so benign overfitting occurs on long-tailed data even when signal is weak.
- Synthetic experiments (K=5, d=1000, m=100): test accuracy increases with noise correlation ratio, and near-optimal accuracy is reached even when feature strength approaches zero.
- On MNIST, CIFAR-10 and CIFAR-100, removing high influence-score samples causes a significantly larger accuracy drop than removing low-score samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the noise correlation ratio | Two-layer CNN on synthetic data · Synthetic data | test accuracy | Helped |
| Increase feature strength | Two-layer CNN on synthetic data · Synthetic data | test accuracy | Helped |
| Use higher noise correlation with feature strength near zero | Two-layer CNN on synthetic data · Synthetic data | test accuracy | Helped |
| Increase dataset size | Two-layer CNN using explicit features · Synthetic data, feature-based classification | test accuracy | Helped |
| Increase dataset size | Two-layer CNN using implicit features · Synthetic data, noise-based classification | test accuracy | No clear effect |
| Remove a portion of examples with high influence scores (ablation) | LeNet, ResNet-18, and ResNet-101 · MNIST, CIFAR-10, and CIFAR-100 | test accuracy | Hurt |
| Remove a portion of examples with low influence scores (ablation) | LeNet, ResNet-18, and ResNet-101 · MNIST, CIFAR-10, and CIFAR-100 | test accuracy | Hurt |
Surprising findings
- Increasing the dataset size of majority classes can hurt the test accuracy of minority classes, since majority class-dependent noise memorization dominates.
- Neural networks can achieve near-optimal test accuracy with near-zero feature strength when the noise correlation ratio is high.
Limitations
- Analysis restricted to two-layer ReLU CNNs trained with full-batch gradient descent and a specific Gaussian-like initialization regime with many over-parameterization conditions.
- Real-world validation is limited to influence-score removal experiments on MNIST and CIFAR, not a direct test of the Gamma-based bound.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100 synthetic feature-noise data
Benchmarks: MNIST CIFAR-10 CIFAR-100 synthetic feature-noise data
Metrics: zero-one test loss (accuracy) training loss test accuracy after removing high/low influence samples
Baselines: Kou et al. 2023 Cao et al. 2022
Builds on: Kou et al. 2023 benign overfitting in two-layer CNNs Cao et al. 2022 benign overfitting feature-noise model feldman memorization in long-tailed data
Closest papers by meaning
- Benign Overfitting in Adversarial Training of Neural Networks (ICML 2024 · similarity 0.91)
- Benign overfitting in leaky ReLU networks with moderate input dimension (NeurIPS 2024 · similarity 0.90)
- Noisy Interpolation Learning with Shallow Univariate ReLU Networks (ICLR 2024 · similarity 0.89)
- Benign Overfitting in Single-Head Attention (NeurIPS 2025 · similarity 0.88)
- Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data (ICLR 2024 · similarity 0.87)
- On the Over-Memorization During Natural, Robust and Catastrophic Overfitting (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.