Research map / Learning theory and algorithms / Kernels and regression theory
Prediction Risk and Estimation Risk of the Ridgeless Least Squares Estimator under General Assumptions on Regression Errors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives exact finite-sample prediction and estimation risk of the minimum-norm ridgeless least squares estimator under correlated, heteroskedastic regression errors, showing the variance terms depend on the error covariance only through its trace.
Problem
Existing benign-overfitting and double-descent analyses of ridgeless least squares assume i.i.d. errors with common variance, which is unrealistic for time series, panel, and clustered data.
Approach
Under left-spherical symmetry of the design matrix, the expected variance of both prediction and estimation risk factorizes into (trace of error covariance)/n times a term depending only on the design, so off-diagonal error correlations and unequal variances do not affect the variance component beyond the trace.
Key results
- Expected variance of prediction risk equals (Tr(Omega)/n) times E[Tr((X^T X)^dagger Sigma)], so only the trace of the error covariance matters.
- The bias term under isotropic-type priors equals r^2 (1 - n/p), independent of the error structure.
- Asymptotically, estimation risk tends to r^2(1-1/gamma) + s* Tr(Omega)/n, reproducing the double descent shape when the spectrum of Sigma is bounded.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Allow correlated, non-identically distributed regression errors with the same trace of the error covariance | Ridgeless least squares · Prediction risk variance | variance | No clear effect |
| Allow correlated, non-identically distributed regression errors with the same trace of the error covariance | Ridgeless least squares · Estimation risk variance | variance | No clear effect |
| Increase the clustered-noise share while keeping overall error variance fixed | Ridgeless least squares · ACS 2018 grouped wage prediction | MSE | No clear effect |
| Increase the overparameterization ratio in the regime p/n > 1 | Ridgeless least squares · Asymptotic estimation risk | estimation risk | Mixed |
Surprising findings
- Off-diagonal elements of the error covariance (e.g., AR(1) or clustered correlation) do not affect the expected variance beyond their contribution to the trace.
Limitations
- Requires left-spherical symmetry of the design matrix, which is stronger than needed in practice and assumed for exact results.
- Results are finite-sample exact only for the variance under the stated assumptions; bias closed forms rely on an isotropic-type prior on beta.
- Conjecture that results hold approximately for non-spherical designs is not formally proven.
Experimental setup
Datasets: American Community Survey 2018 (IPUMS USA) extract
Benchmarks: ACS 2018 group-level wage regression (in-sample vs out-of-sample MSE)
Metrics: mean squared error expected variance of prediction risk expected variance of estimation risk
Builds on: benign overfitting double descent of ridgeless regression Marchenko-Pastur law
Closest papers by meaning
- Risk and cross validation in ridge regression with correlated samples (ICML 2025 · similarity 0.89)
- An Agnostic View on the Cost of Overfitting in (Kernel) Ridge Regression (ICLR 2024 · similarity 0.86)
- Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionality (NeurIPS 2024 · similarity 0.85)
- Benign Overfitting in Out-of-Distribution Generalization of Linear Models (ICLR 2025 · similarity 0.84)
- Beyond Benign Overfitting in Nadaraya-Watson Interpolators (NeurIPS 2025 · similarity 0.84)
- Minimum-Norm Interpolation Under Covariate Shift (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.