Research map / Time series / Causal and generative time series
Risk and cross validation in ridge regression with correlated samples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives sharp high-dimensional asymptotics for ridge regression with correlated training samples via free probability and proposes CorrGCV, an asymptotically exact out-of-sample risk estimator for correlated data.
Problem
Existing high-dimensional ridge regression theory and GCV-based risk estimators assume i.i.d. training examples; with correlated (e.g., time series) samples, these fail to predict out-of-sample risk and hyperparameter tuning is biased.
Approach
Uses random matrix theory and free probability (S-transform, deterministic equivalents) for matrix-Gaussian data with anisotropic feature and sample correlations, giving sharp in- and out-of-sample risk formulas and a corrected GCV (CorrGCV) that is asymptotically exact when noise shares the sample correlation; also extends to test points correlated with training data.
Key results
- CorrGCV accurately predicts out-of-sample risk for exponentially correlated data, while naive GCV variants under- or overestimate it.
- Sample correlations do not change the power-law scaling of risk with sample size T (R_g ~ T^{-2 alpha min(r,1)}), only the constants.
- Correlations mollify the double descent peak when noise is matched to covariate correlation, and exacerbate it when noise is uncorrelated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CorrGCV instead of ordinary GCV | Ridge regression with correlated samples · Correlated samples with matched covariate and noise correlations | out-of-sample risk prediction accuracy | Helped |
| use CorrGCV instead of Naive GCV₁ | Ridge regression · Strongly correlated samples | out-of-sample risk prediction accuracy | Helped |
| use CorrGCV instead of Naive GCV₂ | Ridge regression · Strongly correlated samples | out-of-sample risk prediction accuracy | Helped |
| account for correlation between the held-out point and training samples | CorrGCV for ridge regression · Time-series forecasting with a correlated test point | out-of-sample risk prediction | Helped |
| test on nearby, correlated samples rather than an uncorrelated test point | Ridge regression risk evaluation · Time-series forecasting | out-of-sample risk | Mixed |
| match noise correlations to covariate correlations rather than using uncorrelated noise | Ridge regression · Correlated samples in the overparameterized regime | generalization error | Helped |
| introduce strong sample correlations | Ridge regression with matched noise correlations · Double descent | double-descent peak magnitude | Helped |
| introduce strong sample correlations while keeping noise uncorrelated | Ridge regression · Double descent | double-descent peak magnitude | Hurt |
| introduce sample correlations | Ridge regression · Power-law feature spectra and stationary correlated samples | generalization-error decay rate | No clear effect |
Surprising findings
- Previously proposed GCV corrections for correlated samples are asymptotically biased in high dimensions.
- Uncorrelated noise with correlated covariates is generally worse than matched-correlation noise, magnifying double descent.
Limitations
- Analysis assumes Gaussian (matrix-Gaussian) data; universality for non-Gaussian correlated data is not rigorously established.
- CorrGCV requires known or estimated S-transform of the sample correlation matrix K; without it the estimator is unavailable.
- Results are asymptotic in the proportional limit N, T to infinity.
Experimental setup
Datasets: synthetic Gaussian matrix data with exponential correlations power-law correlated synthetic data
Metrics: out-of-sample risk in-sample risk estimator bias relative to omniscient risk
Baselines: Naive GCV KARE ordinary GCV
Builds on: generalized cross-validation kernel alignment risk estimator free probability S-transform
Closest papers by meaning
- Asymptotically Free Sketched Ridge Ensembles: Risks, Cross-Validation, and Tuning (ICLR 2024 · similarity 0.90)
- Prediction Risk and Estimation Risk of the Ridgeless Least Squares Estimator under General Assumptions on Regression Errors (ICLR 2025 · similarity 0.89)
- Optimal Ridge Regularization for Out-of-Distribution Prediction (ICML 2024 · similarity 0.87)
- An Agnostic View on the Cost of Overfitting in (Kernel) Ridge Regression (ICLR 2024 · similarity 0.86)
- Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionality (NeurIPS 2024 · similarity 0.86)
- Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.