Research map / Label-efficient and robust learning / Domain adaptation and generalization
Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes OLS out-of-distribution risk under covariate shift via eigenspectra and proposes SpAR, a post-hoc method that projects out last-layer regressor components along test eigenvectors where spectral inflation occurs.
Problem
Regression models, including neural regressors, degrade under covariate shift, and OOD generalization for continuous targets is comparatively under-studied relative to classification.
Approach
Derives a closed-form OOD risk for OLS in terms of train and test eigenspectra ("spectral inflation"), then adapts the last-layer weights by projecting out test eigenvectors whose estimated variance loss exceeds bias loss, using unlabeled test inputs.
Key results
- On PovertyMap-WILDS, C-Mixup + SpAR reaches worst-group Spearman r 0.515 vs 0.489 for C-Mixup, and matches the best unlabeled-data worst-group result (0.52 in the comparison table).
- On RCF-MNIST, SpAR improves ERM, C-Mixup, Deep CORAL and DANN RMSE on every backbone (e.g., DANN 0.177 to 0.170).
- On synthetic Experiment 1, ERM+SpAR matches PCR at 1.59e5 squared error versus 2.54e6 for ERM, while on Experiment 4 (no shift) SpAR leaves ERM unchanged.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SpAR to ERM | ERM · Synthetic Experiments 1 and 3 | squared error: 2.54e6 $\pm$ 3.84e6 -> 1.59e5 $\pm$ 3.13e3; 2.54e6 $\pm$ 3.84e6 -> 1.27e5 $\pm$ 2.51e3 | Helped |
| Apply SpAR to ERM | ERM · Synthetic Experiment 2 | squared error: 2.54e6 $\pm$ 3.84e6 -> 2.81e0$\pm$4.53e0 | Helped |
| Apply SpAR to ERM | ERM · Synthetic Experiment 4 | squared error: \textbf{1.68e0} $\pm$ 1.15e0 -> \textbf{1.68e0} $\pm$ 1.15e0 | No clear effect |
| Apply SpAR to ERM, C-Mixup, Deep CORAL, and DANN | Image regression methods · RCF-MNIST | OOD RMSE: 0.155 $\pm$ 0.006 -> 0.154 $\pm$ 0.006; 0.158 $\pm$ 0.011 -> 0.156 $\pm$ 0.009; 0.167 $\pm$ 0.012 -> 0.165 $\pm$ 0.010; | Helped |
| Apply SpAR to ERM, C-Mixup, Deep CORAL, and DANN | Image regression methods · ChairAngles-Tails | OOD RMSE: 6.788 $\pm$ 0.634 -> 6.753 $\pm$ 0.648; 6.504 $\pm$ 0.324 -> 6.449 $\pm$ 0.325; 5.978 $\pm$ 0.243 -> 5.839 $\pm$ 0.259; | Helped |
| Apply SpAR to ERM | ERM · PovertyMap-WILDS | r_all: 0.793 $\pm$ 0.040 -> \textbf{0.794} $\pm$ 0.046 | Helped |
| Apply SpAR to ERM | ERM · PovertyMap-WILDS | r_wg: 0.497 $\pm$ 0.099 -> \textbf{0.512} $\pm$ 0.092 | Helped |
| Apply SpAR to C-Mixup | C-Mixup · PovertyMap-WILDS | r_all: 0.784 $\pm$ 0.045 -> \textbf{0.794} $\pm$ 0.043 | Helped |
| Apply SpAR to C-Mixup | C-Mixup · PovertyMap-WILDS | r_wg: 0.489 $\pm$ 0.045 -> \textbf{0.515} $\pm$ 0.091 | Helped |
Surprising findings
- Improvements from SpAR on image benchmarks are small (roughly 0.5% to 4% RMSE reduction), and on PovertyMap the average-group r is essentially unchanged.
- Plain pseudoinverse (ERM + OLS) performs worse than the gradient-trained ERM on tabular benchmarks, showing that naive OLS replacement of the last layer can hurt.
Limitations
- Assumes access to unlabeled test inputs and covariate shift only.
- Theory relies on linear OLS analysis and a Gaussian noise assumption; applied to gradient-trained neural networks via heuristic.
- Requires SVD of train and test representations, and the rejection confidence alpha was fixed at 0.999 with limited tuning.
Experimental setup
Datasets: CommunitiesAndCrime SkillCraft RCF-MNIST ChairAngles-Tails PovertyMap-WILDS synthetic 2D Gaussian data
Benchmarks: CommunitiesAndCrime SkillCraft RCF-MNIST ChairAngles-Tails PovertyMap-WILDS synthetic 2D Gaussian data
Metrics: OOD RMSE average RMSE worst group RMSE Spearman r (all and worst group)
Baselines: ERM C-Mixup PCR DANN Deep CORAL Noisy Student SwAV AFN RSD DARE-GRAM
Builds on: Ordinary Least Squares Principal Component Regression C-Mixup DANN Deep CORAL
Closest papers by meaning
- Sharp Generalization for Nonparametric Regression by Over-Parameterized Neural Networks: A Distribution-Free Analysis in Spherical Covariate (ICML 2025 · similarity 0.87)
- Improving Adaptivity via Over-Parameterization in Sequence Models (NeurIPS 2024 · similarity 0.86)
- Free Decompression with Algebraic Spectral Curves (NeurIPS 2026 · similarity 0.86)
- Computational Efficiency under Covariate Shift in Kernel Ridge Regression (NeurIPS 2025 · similarity 0.86)
- High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularization (ICML 2024 · similarity 0.86)
- Generalization in Kernel Regression Under Realistic Assumptions (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.