Research map / Learning theory and algorithms / Kernels and regression theory
No Double Descent in Principal Component Regression: A High-Dimensional Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives asymptotic random-matrix-theory risk formulas for principal component regression on spiked-covariance latent factor data, including covariate shift, and shows PCR avoids the interpolation peak of double descent.
Problem
Generalization risk of principal component regression in high dimensions when data lies near a low-dimensional subspace, and how PCR behaves under the double descent phenomenon and train-test distribution shift.
Approach
Extends random-matrix-theory results for spiked covariance models from PCA eigenvalues/eigenvectors to the supervised PCR setting, yielding asymptotic bias-variance risk expressions for PCR with and without covariate shift in a latent factor model.
Key results
- Asymptotic PCR risk (Theorem 2) matches finite-sample simulations across gamma in [0.1, 30] with n=500, d=10 spikes.
- PCR with appropriately chosen k (k near d) avoids the interpolation peak at gamma=1 that full linear regression exhibits.
- Choosing k too small (k<d) causes bias to rise sharply; k much larger than d pushes PCR toward full regression risk.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PCR with appropriately chosen k (k = d+Δ) instead of full regression | Linear regression on spiked covariance data · Simulation, in-distribution spiked covariance | risk near interpolation peak (γ=1) | Helped |
| Choose k too small (k < d) in PCR (ablation) | PCR on spiked covariance data · Simulation, in-distribution spiked covariance | risk | Hurt |
| Choose k much larger than d (k ≫ d) in PCR (ablation) | PCR on spiked covariance data · Simulation, in-distribution spiked covariance | regularizing effect of PCR | Hurt |
| Use highly correlated train/test latent factors (ρ=0.9) instead of low correlation (ρ=0.1) (ablation) | PCR under covariate shift · Simulation, covariate-shifted spiked covariance | risk | Helped |
| Set k to be small (γ<0.5 or γ>2 regime) so that k has little effect | PCR choice of k · Simulation, in-distribution spiked covariance | risk sensitivity to k | No clear effect |
Limitations
- Analysis limited to linear latent factor model with identity base covariance and asymptotic regime only
- Finite-sample guarantees not provided (referred to Bing et al. 2021)
- Extension to gradient-based training or general covariance not covered
Experimental setup
Datasets: Diverse MAGIC wheat genetics data set (motivating example only)
Metrics: expected test risk (MSE) squared bias variance
Baselines: full linear regression null predictor
Builds on: Hastie et al. 2022 linear regression double descent analysis Johnstone 2001 spiked covariance model Marchenko-Pastur theorem Baik et al. 2005 eigenvalue phase transition Emami et al. 2020 covariate shift model
Closest papers by meaning
- On Robustness of Principal Component Regression (NeurIPS 2019 · similarity 0.91)
- Faster Principal Component Regression and Stable Matrix Chebyshev Approximation (ICML 2017 · similarity 0.87)
- Capturing the denoising effect of PCA via compression ratio (NeurIPS 2024 · similarity 0.84)
- Risk and cross validation in ridge regression with correlated samples (ICML 2025 · similarity 0.84)
- Least Squares Regression Can Exhibit Under-Parameterized Double Descent (NeurIPS 2024 · similarity 0.84)
- High-Dimensional Tensor Regression With Oracle Properties (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.