Research map / Label-efficient and robust learning / OOD and anomaly detection
Benign Overfitting in Out-of-Distribution Generalization of Linear Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives non-asymptotic benign-overfitting guarantees for over-parameterized ridge regression under general covariate shift, and shows PCR achieves O(1/n) where ridge is stuck at O(1/sqrt(n)) under large minor-direction shift.
Problem
Whether benign overfitting of over-parameterized linear models (ridge / minimum-norm interpolation) persists under covariate shift between source and target distributions, and how it compares to alternatives like PCR when the target covariance has large minor-direction components.
Approach
Derives the first vanishing, non-asymptotic excess-risk bound for ridge regression under arbitrary target covariance, showing only the overall magnitude (trace/norm) of target minor-direction covariance matters rather than its spectrum; for large minor shift, shows ridge has an O(1/sqrt(n)) lower bound while PCR attains O(1/n).
Key results
- Ridge (ridgeless) under covariate shift shows excess risk decaying with slope near -1 in simulations, confirming O(1/n) benign overfitting when minor shift is controlled.
- Under large minor-direction shift (Sigma_T = I_d), ridge's optimal rate is O(n^-0.48) with lambda = n^0.75, consistent with the theoretical O(1/sqrt(n)) lower bound.
- PCR achieves log-log slope -0.99 (near O(1/n)) in the same large-shift setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Control the target covariance magnitude in the minor directions | Over-parameterized ridge regression · OOD covariate shift | excess risk: \mathcal O(1/n) | Helped |
| Increase the target covariance magnitude in the minor directions | Ridge regression · OOD covariate shift with large minor-direction shift | excess risk: \mathcal O(1/\sqrt{n}) | Hurt |
| Regress only on the estimated top-k principal-component subspace | Linear model under large minor-direction shift · OOD covariate shift | excess risk: \mathcal{O}(1/n) versus \mathcal O(1/\sqrt{n}) | Helped |
| Vary the major-direction shift while holding the minor-direction shift ratio at 1 | Minimum norm interpolation · OOD covariate shift, small shift in minor directions | excess risk | Hurt |
| Vary the minor-direction shift ratio while holding the major-direction shift at 1 | Minimum norm interpolation · OOD covariate shift, small shift in minor directions | excess risk | Hurt |
| Use PCR instead of ridge regression | Linear model under large minor-direction shift · OOD covariate shift | excess risk slope: -0.48 -> -0.99 | Helped |
Surprising findings
- Only the overall magnitude of target minor-direction covariance matters for benign overfitting, not its spectral structure, while the source minor directions must have high effective rank.
- Ridge regression incurs constant excess risk for certain regularization choices under large minor shift, even though it achieves O(1/n) in-distribution.
Limitations
- Lower bound for ridge only shown for specific instances; no general matching lower bound.
- Analysis restricted to linear models; no nonlinear extension.
- Assumes CondNum condition and sub-gaussian source covariates with large effective rank of minor directions.
Experimental setup
Benchmarks: simulated Gaussian linear regression
Metrics: excess risk log-log slope vs n
Baselines: in-distribution ridge benign overfitting bound ridge regression ridgeless regression
Builds on: benign overfitting (Bartlett et al. 2020) sharp covariate shift analysis for MLE (Kpotufe/Martinet-style Fisher informatio CondNum assumption
Closest papers by meaning
- An Agnostic View on the Cost of Overfitting in (Kernel) Ridge Regression (ICLR 2024 · similarity 0.88)
- Benign Overfitting in Adversarial Training of Neural Networks (ICML 2024 · similarity 0.86)
- Minimum-Norm Interpolation Under Covariate Shift (ICML 2024 · similarity 0.86)
- Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression (NeurIPS 2025 · similarity 0.86)
- Rethinking Benign Overfitting in Two-Layer Neural Networks (ICML 2025 · similarity 0.86)
- Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrum (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.