Research map / Label-efficient and robust learning / Noisy labels
Unraveling the Enigma of Double Descent: An In-depth Analysis through the Lens of Learned Feature Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Replicates model-wise double descent on FCNNs, CNNs and ResNet18s with injected label noise and uses a cosine-similarity k-NN in the penultimate feature space to argue that over-parameterized models "isolate" noisy labels among clean same-class samples.
Problem
The mechanism behind deep double descent is not established; the paper asks how over-parameterized networks interpolate noisy labels and whether this relates to improved generalization.
Approach
Uses a k-nearest-neighbour probe (cosine similarity) on penultimate-layer features to measure, for each mislabeled training point, whether its nearest clean neighbours share its original class (the P metric), and correlates P with the double descent test-error curve across model widths.
Key results
- On MNIST (N=4000) two-layer FCNN, with 10%/20% label noise the test error shows a clear peak at the interpolation threshold; P rises to nearly 100% and test accuracy to about 85% past the threshold.
- On CIFAR-10 CNNs, P rises to about 60% for over-parameterized models after the test-error peak, lower than on MNIST.
- For ResNet18 the k-NN prediction on noisy labels goes to the noisy labels for large models, contradicting the isolation hypothesis for that architecture.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Introduce label noise in FCNN training | FCNN on MNIST · MNIST | test error | Mixed |
| Increase label-noise ratio | FCNN on MNIST · MNIST | test error peak | Hurt |
| Increase FCNN width beyond the interpolation threshold | FCNN on MNIST · MNIST | test accuracy | Helped |
| Increase FCNN width beyond the interpolation threshold | FCNN on MNIST · MNIST noisy-label prediction | k-NN prediction accuracy P: to 85% and nearly 100% | Helped |
| Introduce label noise | CNN on CIFAR-10 · CIFAR-10 | test error peak | Hurt |
| Increase CNN width | CNN on CIFAR-10 · CIFAR-10 noisy-label prediction | k-NN prediction accuracy P: to about 60% | Mixed |
| Increase CNN width into the over-parameterized region | CNN on CIFAR-10 · CIFAR-10 noisy-label prediction | k-NN prediction accuracy P: to about 60% | Helped |
| Increase ResNet18 width beyond the interpolation threshold | ResNet18 on CIFAR-10 · CIFAR-10 noisy-label prediction | k-NN prediction accuracy on clean labels: to zero | Hurt |
| Use an over-parameterized CNN instead of a ResNet at higher label noise | CNN and ResNet18 on CIFAR-10 · CIFAR-10 | test accuracy: by 5% | Helped |
| Introduce high label noise | Models trained on MNIST and CIFAR-10 · MNIST and CIFAR-10, 40% label noise | test accuracy | Hurt |
| Introduce high label noise | Models trained on MNIST and CIFAR-10 · MNIST and CIFAR-10, 40% label noise | correlation between P and generalization performance | No clear effect |
Surprising findings
- For ResNet18s, over-parameterized models' learned representations of noisy samples align with their noisy labels, contradicting the isolation hypothesis.
- The k-NN probe P is said to be only a weak predictor of generalization, since it does not use unseen data.
Limitations
- The k-NN analysis is only an estimation technique with cosine similarity, and is less informative for CNN feature spaces.
- The ResNet18 results contradict the main isolation hypothesis, leaving the mechanism incompletely established.
- No formal theory is developed; the mechanism is argued qualitatively.
Experimental setup
Datasets: MNIST CIFAR-10
Benchmarks: MNIST CIFAR-10
Metrics: test error test accuracy train error k-NN prediction accuracy P on noisy labels
Baselines: Nakkiran et al. deep double descent experiments Bartlett et al. benign overfitting Belkin et al. double descent
Builds on: deep double descent (Nakkiran et al.) benign overfitting bias-variance trade-off
Closest papers by meaning
- On the Role of Optimization in Double Descent: A Least Squares Study (NeurIPS 2021 · similarity 0.93)
- A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning (NeurIPS 2023 · similarity 0.92)
- DSD²: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free? (AAAI 2024 · similarity 0.88)
- Double Descent Meets Out-of-Distribution Detection: Theoretical Insights and Empirical Analysis on the Role of Model Complexity (NeurIPS 2025 · similarity 0.87)
- Least Squares Regression Can Exhibit Under-Parameterized Double Descent (NeurIPS 2024 · similarity 0.86)
- Double-Descent Curves in Neural Networks: A New Perspective Using Gaussian Processes (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.