Research map / Optimization / Implicit bias and generalization
How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes label noise gradient descent (random label flipping at each step) for a two-layer squared-ReLU CNN on a signal-noise model, proving that it suppresses noise memorization and generalizes in low SNR where standard GD overfits to noise.
Problem
Gradient descent on neural networks overfits to noise and generalizes poorly when the signal-to-noise ratio is low; existing fixes (graph smoothing, SAM) have restrictive conditions or extra cost.
Approach
Inject random label flipping (probability p per sample per step) into ordinary gradient descent with no extra computation; the flipping regularizes noise memorization to a constant level while signal learning keeps growing, yielding provable generalization in the low SNR regime.
Key results
- Theory: standard GD reaches training loss <= epsilon but test error >= 0.24 when SNR = O(n^{-1/2}); label noise GD keeps training loss Theta(1) while test error <= 2exp(-Cd/n^2).
- Synthetic experiments (n=200, d=2000, p=0.1): standard GD fits training loss near zero with noise memorization exceeding signal learning, while label noise GD steadily improves test accuracy.
- VGG-16 on CIFAR-10 with added high-frequency noise: label noise GD outperforms standard GD, with the gap growing as SNR decreases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add random label flips with probability p at each iteration | Gradient descent on a two-layer CNN with squared ReLU · Low-SNR signal-noise model | test error: ≥ 0.24 vs ≤ 2 \exp \left( - \frac{Cd}{ n^2 } \right) | Helped |
| Add random label flips with probability p at each iteration | Gradient descent on a two-layer CNN with squared ReLU · Low-SNR signal-noise model | training loss: ≤ \epsilon vs = {\Theta(1)} | Mixed |
| Add random label flips with probability 0.1 at each iteration | Two-layer squared-ReLU network · Synthetic signal-noise data | test accuracy | Helped |
| Add random label flips with probability 0.1 at each iteration | Two-layer squared-ReLU network · Synthetic signal-noise data | training loss | Mixed |
| Add random label flips during training | Neural network trained with GD · Low-SNR regimes across SNR and sample sizes | test accuracy | Helped |
Surprising findings
- Label noise GD intentionally keeps training loss at a constant order (oscillating around 0.5) yet generalizes better than GD that drives training loss to near zero.
Limitations
- Theory restricted to two-layer CNN with squared ReLU activation and full-batch GD.
- Analysis does not extend to SGD or adaptive optimizers like Adam.
- Label noise GD requires more iterations than standard GD to reach good test performance (stopping-time ratio Theta(n SNR^2) under the low SNR assumption).
Experimental setup
Datasets: CIFAR-10 synthetic signal-noise data
Benchmarks: CIFAR-10 synthetic signal-noise data
Metrics: test accuracy 0-1 test loss training logistic loss
Baselines: standard GD SAM SGD
Builds on: signal-noise data model feature learning theory sharpness-aware minimization label noise SGD
Closest papers by meaning
- Noise against noise: stochastic label noise helps combat inherent label noise (ICLR 2021 · similarity 0.91)
- On the Role of Label Noise in the Feature Learning Process (ICML 2025 · similarity 0.90)
- Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks (ICLR 2024 · similarity 0.88)
- Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic Optimization (NeurIPS 2025 · similarity 0.87)
- Mitigating Label Noise through Data Ambiguation (AAAI 2024 · similarity 0.87)
- Learning with Structural Labels for Learning with Noisy Labels (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.