Research map / Learning theory and algorithms / Kernels and regression theory
Optimal Regularization can Mitigate Double Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Tune ℓ2 regularization optimally | Isotropic well-specified linear regression · Isotropic Gaussian regression, increasing samples | expected test risk | Helped |
| Use optimal ℓ2 regularization as model size grows | Linear regression on random projected features · Isotropic random-projection regression, increasing model size | expected test risk | Helped |
| Use optimal ℓ2 regularization | Ridge regression · Non-isotropic Gaussian regression, increasing samples | expected test risk | Helped |
| Replace the standard penalty with a covariance-dependent Σ⁻¹ penalty and tune its strength | Ridge regression · Non-isotropic Gaussian regression, increasing samples | expected test risk | Helped |
| Tune ℓ2 regularization | Random ReLU features · Fashion-MNIST, increasing training samples | test classification error | Helped |
| Tune ℓ2 regularization | Random ReLU features · Fashion-MNIST, increasing feature count | test classification error | Helped |
| Tune weight decay for each model size | 5-layer CNN · CIFAR-100, increasing model width | test error | Helped |
| Use optimally tuned ridge regularization as samples increase | Ridge regression with non-Gaussian covariates and heteroscedastic noise · Two-dimensional counterexample, n=1 versus n=2 | expected test risk: < 8.157 -> > 8.179 | Hurt |
| Increase sample count while holding the ridge parameter at or above its optimum | Isotropic well-specified linear regression · Isotropic Gaussian regression, increasing samples | expected test risk | Helped |
Closest papers by meaning
- On the Role of Optimization in Double Descent: A Least Squares Study (NeurIPS 2021 · similarity 0.92)
- Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel (NeurIPS 2019 · similarity 0.89)
- Double Trouble in Double Descent: Bias and Variance(s) in the Lazy Regime (ICML 2020 · similarity 0.88)
- Unraveling the Enigma of Double Descent: An In-depth Analysis through the Lens of Learned Feature Space (ICLR 2024 · similarity 0.87)
- DSD²: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free? (AAAI 2024 · similarity 0.87)
- Least Squares Regression Can Exhibit Under-Parameterized Double Descent (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.