Research map / Label-efficient and robust learning / Noisy labels
On the Role of Label Noise in the Feature Learning Process
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically analyzes two-layer ReLU CNN training dynamics under label noise on a signal-noise data model, identifying a two-stage process (clean-sample fitting then noise memorization) that justifies early stopping and small-loss sample selection.
Problem
How label noise affects the feature learning process of neural networks beyond the lazy training regime, including why generalization degrades over training.
Approach
Analyzes gradient descent on a two-layer ReLU CNN under a signal-noise data distribution with random label flipping, under a constant n·SNR² regime, showing that signal learning dominates early (fitting clean samples, ignoring noisy ones) and noise memorization overtakes later (overfitting noisy labels), which theoretically grounds early stopping and small-loss sample selection.
Key results
- Theory: in Stage I all clean samples are fit and noisy samples are classified to ground truth; in Stage II the test error is at least 0.5·min{τ+, τ−} under label noise, while without noise test error is bounded by exp(d/n − n‖μ‖⁴/(C_D σ_ξ⁴ d)).
- Synthetic experiments (n=100, d=2000, 10% flip, n·SNR²=20) show max signal coefficient dominating early and noise coefficients on noisy samples overtaking later, with noisy-sample accuracy first dropping to 0 then rising.
- Real-world VGG on CIFAR-10 (first two classes, 20% flip) shows accuracy on noisy samples dropping then rising to 1 and test accuracy lower than the no-noise run.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with label noise until the training loss converges | Two-layer ReLU CNN on the signal-noise distribution · Signal-noise distribution with label noise, Stage II | test error | Hurt |
| stop training at T_1 during Stage I instead of continuing until loss convergence | Two-layer ReLU CNN on the signal-noise distribution with label noise · Signal-noise distribution with label noise, Stage I | test error | Helped |
| remove label flipping | Two-layer ReLU CNN on the signal-noise distribution · Signal-noise distribution without label noise | test error | Helped |
| train with label flipping rather than without label flipping | VGG net on CIFAR-10 first two categories · CIFAR-10 test set | accuracy | Hurt |
| train with label noise | Two-layer CNN on the synthetic signal-noise setup · Synthetic noisy samples during training | accuracy: 0 to 1 | Mixed |
| select samples using the small-loss criterion at T_1 | Two-layer ReLU CNN on the signal-noise distribution with label noise · Signal-noise distribution with label noise, Stage I | clean/noisy sample separation: \log(2) | Helped |
Surprising findings
- Noisy samples are initially classified to their ground-truth class during Stage I, so training accuracy on noisy samples first drops toward zero before memorization raises it.
Limitations
- Analysis limited to random (class-independent) label noise, not data-dependent or adversarial noise.
- Theory restricted to two-layer CNN; unclear if two-stage dynamics persist in deeper networks with residuals or normalization.
- Requires strong conditions on dimension, width, sample size, SNR and learning rate.
Experimental setup
Datasets: synthetic signal-noise data CIFAR-10
Benchmarks: CIFAR-10 (two-class subset) synthetic signal-noise test distribution
Metrics: 0-1 test error training accuracy on clean and noisy samples signal and noise coefficient magnitudes
Baselines: Lazy-regime label noise analyses (constrained-weight / NTK) Prior feature learning analyses without label noise
Builds on: feature learning theory signal-noise model (Allen-Zhu & Li style CNN analysis) small-loss criterion early stopping
Closest papers by meaning
- Understanding Instance-Level Label Noise: Disparate Impacts and Treatments (ICML 2021 · similarity 0.90)
- On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD (AAAI 2026 · similarity 0.90)
- How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime? (NeurIPS 2025 · similarity 0.90)
- Learning with Structural Labels for Learning with Noisy Labels (CVPR 2024 · similarity 0.89)
- Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks (ICLR 2024 · similarity 0.89)
- Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic Optimization (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.