Research map / Optimization / Implicit bias and generalization
Least Squares Regression Can Exhibit Under-Parameterized Double Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows with two simple linear models (ridge spiked-covariance and a Gaussian/rank-one mixture) that double descent peaks can occur strictly inside the under-parameterized regime, challenging the view that the peak lies at the interpolation point.
Problem
Prior theory places double descent peaks at the interpolation threshold or in the over-parameterized regime, and holds that the variance is monotone for under-parameterized linear regression; the paper asks whether peaks can occur under-parameterized and what causes peak location.
Approach
Peak location depends on the alignment between targets and right singular vectors of the training matrix and on the spectrum's Stieltjes transform at zero; violating either assumption (via ridge-induced misalignment or a low-rank mixture spectrum) moves the peak to c = 1/(1+mu^2) or c = pi_1 inside the under-parameterized regime.
Key results
- For the ridge spiked-covariance model, the risk has a local maximum at c = 1/(1+mu^2) < 1, verified numerically against theory.
- For the Gaussian plus rank-one mixture model, the only peak in the risk occurs at c = pi_1 in the under-parameterized regime, with excess risk pi_1 c/(pi_1 - c) for c < pi_1.
- In the parameter scaling regime, the estimator norm can show a peak under-parameterized even when the risk curve is monotonic for larger mu.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ridge regularization in the rank-1 signal model with input noise | Least squares alignment mismatch model · Alignment mismatch model (data scaling regime) | generalization risk | Mixed |
| Mix isotropic Gaussian samples with samples along a fixed direction | Minimum-norm least squares · Mixture model | expected excess risk | Helped |
| Modify the noise spectrum while leaving the problem unregularized (ablation) | Alignment mismatch model · Alignment mismatch ablation | generalization risk | No clear effect |
| Replace the right singular vectors with a uniformly random orthogonal matrix (ablation) | Alignment mismatch model · Alignment mismatch ablation | generalization risk | Mixed |
| Increase ridge regularization while fixing training-set size and varying dimension | Alignment mismatch model · Parameter scaling regime | generalization risk | Hurt |
| Increase ridge regularization while fixing training-set size and varying dimension | Alignment mismatch model · Parameter scaling regime | estimator norm | No clear effect |
| Project MNIST to a low-dimensional subspace before adding noise | Linear denoiser · Low-dimensional MNIST denoising | generalization error | Mixed |
| Remove the low-dimensional projection before denoising (ablation) | Linear denoiser · MNIST denoising, 784 dimensions | generalization error | Hurt |
| Use a one-hidden-layer neural network on noisy data projected to a random 10-dimensional subspace | MNIST denoising network · Low-dimensional MNIST denoising | generalization error: 4 peaks | Mixed |
| Enforce β perpendicular to the spike direction (ablation) | Minimum-norm least squares mixture model · Mixture model | expected excess risk | No clear effect |
| Optimize training signal strength instead of using a fixed value (ablation) | Ridge-regularized least squares · Alignment mismatch model | generalization error | No clear effect |
| Use higher-quality training data while increasing ridge regularization to compensate | Ridge-regularized least squares · Alignment mismatch model | generalization error | Helped |
Surprising findings
- The risk can have its only peak in the under-parameterized regime, contrary to prior theories placing peaks at the interpolation point.
- Peak in the norm of the estimator does not imply a peak in the generalization risk for large ridge parameter.
Limitations
- Results are for linear models and rank-one or low-rank data; extension to deep networks is open.
- Some results are asymptotic and the low-rank extension is only conjectured.
- The non-linear MNIST experiment is only qualitative and mechanisms are unclear.
Experimental setup
Datasets: MNIST
Benchmarks: MNIST
Metrics: expected generalization risk training error norm of estimator
Baselines: classical bias-variance trade-off prior linear double descent theory
Builds on: Marchenko-Pastur distribution double descent theory for linear models
Closest papers by meaning
- On the Role of Optimization in Double Descent: A Least Squares Study (NeurIPS 2021 · similarity 0.89)
- Multiple Descent: Design Your Own Generalization Curve (NeurIPS 2021 · similarity 0.88)
- Unraveling the Enigma of Double Descent: An In-depth Analysis through the Lens of Learned Feature Space (ICLR 2024 · similarity 0.86)
- Double-Descent Curves in Neural Networks: A New Perspective Using Gaussian Processes (AAAI 2024 · similarity 0.85)
- No Double Descent in Principal Component Regression: A High-Dimensional Analysis (ICML 2024 · similarity 0.84)
- Prediction Risk and Estimation Risk of the Ridgeless Least Squares Estimator under General Assumptions on Regression Errors (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.