Research map / Learning theory and algorithms / Kernels and regression theory

Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization

ICML 2025 · Learning theory and algorithms › Kernels and regression theory · approach: High-dimensional linear regression theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Characterizes spurious correlations learned by ridge regression in high dimensions via a deterministic covariance-based quantity, links it to Schur complement and spurious-feature simplicity, shows a regularization trade-off with test loss, and proves random features equal regularized linear regression.

Problem

How much ERM-learned linear and random-features regression models rely on spurious features correlated with labels in high-dimensional data, and how this depends on ridge regularization, feature simplicity, and over-parameterization.

Approach

Defines a covariance-based spurious correlation measure C for regression, shows for ridge regression it concentrates around a deterministic quantity C^Σ(λ) expressed via the data covariance and an effective regularizer τ(λ), bounds it using the Schur complement and spectrum of the spurious covariance, and proves random-features models are asymptotically equivalent to ridge regression with an effec

Key results

• Proves E_E[C]=0 for unregularized linear regression and |C| = O(log d / sqrt d) w.h.p. when n = omega(d).
• Shows the optimal regularization minimizing test loss lies in an interval (0, lambda_C] where spurious correlation C is positive and increasing, under condition 2d/n bound.
• Proves random features model with p = omega(n log^4 n) is asymptotically equivalent to ridge regression with effective regularization tilde-lambda = 2 tilde-mu^2 d/(mu_1^2 n) + 2d lambda/(mu_1^2 p), so ridgeless RF has positive effective regularization.

Measured effects reported in the paper

Surprising findings

• Unregularized linear regression learns no spurious correlation, yet ridgeless random features (e.g., tanh) do, because of an implicit effective regularization.
• Spurious correlations can reduce in-distribution test loss, even though they hurt out-of-distribution loss.

Limitations

• Results are asymptotic in the proportional regime n = Theta(d) or require n = omega(d) for the simpler bound; analysis restricted to linear regression and random features.
• Assumes Gaussian data (relaxed to Lipschitz concentration for RF) and specific odd, Lipschitz activation with nonzero first Hermite coefficient.
• Experiments on neural networks are qualitative; the theory does not directly cover deep networks trained by SGD.

Experimental setup

Datasets: Gaussian synthetic data Color-MNIST CIFAR-10

Benchmarks: Gaussian synthetic data Color-MNIST CIFAR-10

Metrics: spurious correlation C in-distribution test loss L out-of-distribution test loss lower bound

Baselines: ridgeless linear regression two-layer neural network trained on Color-MNIST and CIFAR-10

Builds on: convex Gaussian min-max theorem random features model Gaussian sequence model characterization of ridge regression

Closest papers by meaning

• Let Samples Speak: Mitigating Spurious Correlation by Exploiting the Clusterness of Samples (CVPR 2025 · similarity 0.88)
• High-Dimensional Analysis for Generalized Nonlinear Regression: From Asymptotics to Algorithm (AAAI 2024 · similarity 0.88)
• Complexity Matters: Feature Learning in the Presence of Spurious Correlations (ICML 2024 · similarity 0.87)
• Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionality (NeurIPS 2024 · similarity 0.86)
• Risk and cross validation in ridge regression with correlated samples (ICML 2025 · similarity 0.86)
• Dimension-free deterministic equivalents and scaling laws for random feature regression (NeurIPS 2024 · similarity 0.85)