Research map / Learning theory and algorithms / Kernels and regression theory
Optimal Ridge Regularization for Out-of-Distribution Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Characterizes optimal ridge penalty sign and optimally tuned risk monotonicity for out-of-distribution (covariate and regression shift) prediction in proportional asymptotics, allowing negative regularization.
Problem
How the optimal ridge penalty and optimally tuned risk behave in high-dimensional ridge regression when the test distribution shifts arbitrarily (covariate or regression shift) from the train distribution, including negative regularization levels.
Approach
Derives deterministic equivalents for OOD ridge risk under general moment-bounded data with the widest negative-regularization range (via a tighter lower bound lambda_min(phi)), gives general alignment conditions determining the sign of optimal lambda under covariate/regression shift, and extends subsampling-ridge equivalences to negative lambda to prove monotonicity of optimally tuned OOD risk in
Key results
- Under covariate and regression shift, the optimal ridge penalty can be negative even with isotropic features or underparameterized design, unlike the in-distribution case.
- On MNIST, the optimal ridge penalty goes from 1.03 (no shift) to -0.48 under increasing label-exclusion shift.
- The optimally tuned OOD risk is monotonically increasing in the aspect ratio phi (and in SNR under some conditions), even when optimizing over negative lambda.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase covariate shift while keeping the signal isotropic | Optimal ridge regression with random isotropic signals · OOD prediction under covariate shift | optimal ridge penalty λ* | No clear effect |
| Use deterministic rather than isotropic random signals under covariate shift | Optimal ridge regression · OOD prediction under covariate shift, φ<1 | optimal ridge penalty sign | No clear effect |
| Use estimation risk under covariate shift in the overparameterized regime | Optimal ridge regression · Estimation risk, φ>1 | optimal ridge penalty sign | No clear effect |
| Align the test covariance with a deterministic signal under covariate shift | Optimal ridge regression · OOD prediction under covariate shift, φ>1 | optimal ridge penalty sign | Helped |
| Introduce regression shift satisfying the stated general alignment conditions | Optimal ridge regression · OOD prediction under regression shift, φ>1 | optimal ridge penalty sign | Helped |
| Optimize the ridge penalty over its allowed negative range | Optimal ridge regression · OOD prediction across arbitrary covariate and regression shifts | optimally tuned risk versus data aspect ratio | Helped |
| Increase the data aspect ratio while keeping SNR and test noise fixed | Optimally tuned ridge regression · OOD prediction | optimal risk | Hurt |
| Increase SNR while keeping φ, train noise, and test noise fixed | Optimally tuned ridge regression with β=β₀ · OOD prediction | optimal risk | Hurt |
| Use a fixed positive ridge penalty instead of optimal tuning | Ridge regression · In-distribution risk with isotropic design | risk versus data aspect ratio | Mixed |
| Exclude labels 4, then 3 and 4, then 2, 3, and 4 from the MNIST test distribution | Optimal ridge regression · MNIST OOD | optimal ridge penalty λ*: 1.03 -> 0.48 -> 0.00 -> -0.48 | Mixed |
| Exclude labels 1, 2, 3, and 4 from the MNIST test distribution | Optimal ridge regression · MNIST OOD | optimal ridge penalty λ*: −0.48 -> 1.44 | Mixed |
Surprising findings
- Optimal ridge penalty can be negative under covariate or regression shift even when features are isotropic or the design is underparameterized.
- Under isotropic signals, the optimal penalty phi/SNR is invariant to covariate shift.
Limitations
- Results are asymptotic deterministic equivalents under proportional asymptotics, not finite-sample.
- Oracle optimal penalty; no data-dependent tuning procedure is analyzed.
- Analysis restricted to standard ridge regression (kernel ridge and lasso left to future work).
Experimental setup
Datasets: MNIST
Benchmarks: MNIST
Metrics: prediction risk (asymptotic deterministic equivalent) optimal ridge penalty
Baselines: in-distribution optimal ridge (positive lambda) naive negative lambda lower bound
Builds on: random matrix theory deterministic equivalents for ridge regression subsampling and ridge equivalence (ensembling) Wishart minimum eigenvalue bound for negative ridge
Closest papers by meaning
- Generalized equivalences between subsampling and ridge regularization (NeurIPS 2023 · similarity 0.90)
- Risk and cross validation in ridge regression with correlated samples (ICML 2025 · similarity 0.87)
- No Free Lunch from Random Feature Ensembles: Scaling Laws and Near-Optimality Conditions (ICML 2025 · similarity 0.86)
- High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularization (ICML 2024 · similarity 0.86)
- Ridge Boosting is Both Robust and Efficient (NeurIPS 2025 · similarity 0.86)
- Optimal Rates for Vector-Valued Spectral Regularization Learning Algorithms (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.