Research map / Learning theory and algorithms / Sample complexity
Maximum Likelihood Estimation is All You Need for Well-Specified Covariate Shift
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that vanilla MLE on source data is minimax optimal for well-specified parametric covariate shift, with excess risk scaling as Tr(I_T I_S^{-1})/n, and that MWLE is optimal in some misspecified cases.
Problem
Which estimator is most effective for OOD generalization under covariate shift when the parametric model is well-specified, and what is the fundamental (minimax) limit in non-asymptotic regimes?
Approach
Non-asymptotic analysis of source-only MLE under covariate shift gives excess target risk Tr(I_T I_S^{-1})/n without bounded density ratio, and a matching instance-dependent minimax lower bound shows no estimator beats MLE up to constants; under misspecification, MWLE becomes minimax optimal in certain constructed scenarios.
Key results
- MLE excess target risk is bounded by Tr(I_T I_S^{-1}) log(d/delta)/n with probability 1-2delta, without bounded density ratio assumptions.
- A minimax lower bound of 1/(50n) on Tr(I_T I_S^{-1})^{-1} times expected excess risk holds for any estimator, matching MLE's rate.
- For linear regression under mean shift alpha and scale sigma, the bound is (||alpha||^2 + sigma^2 d)/n.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use source-data MLE without importance weighting | Well-specified parametric covariate-shift models · Well-specified parametric covariate shift | target excess risk: R_{\beta^{\star}}(\beta_{{\sf MLE}}) \leq c\frac{{\sf Tr}\left(\mathcal{I}_T\mathcal{I}^{-1}_S\right)\log \frac{d}{\delt | Helped |
| Use MLE instead of MWLE | Well-specified parametric covariate-shift models · Well-specified parametric covariate shift | asymptotic variance: {\sf Tr}(G_w H_w^{-1})\geq {\sf Tr}(\mathcal{I}_T\mathcal{I}_S^{-1}) | Helped |
| Use MLE under model misspecification rather than adapting to the target distribution | Classical MLE · Certain misspecified covariate-shift scenarios | consistency | Hurt |
| Use MWLE with density-ratio weighting | Misspecified parametric covariate-shift models · Certain misspecified scenarios | consistency | Helped |
| Use MLE for linear regression under covariate shift | Linear regression · Well-specified linear regression under covariate shift | target excess risk: c\frac{\left(\|\alpha\|^2_2+\sigma^2 d\right)\log \frac{d}{\delta}}{n} | Helped |
| Use MLE for logistic regression under covariate shift | Logistic regression · Well-specified logistic regression under covariate shift | target excess risk: c\frac{(d+r^2)\log \frac{d}{\delta}}{n} | Helped |
| Use MLE for phase retrieval under covariate shift | Phase retrieval · Well-specified phase retrieval under covariate shift | target excess risk: c\frac{(d+r^2)\log \frac{d}{\delta}}{n} | Helped |
Surprising findings
- Source-only vanilla MLE outperforms importance-weighted MLE in the well-specified setting, contrary to common importance weighting practice.
- Under misspecification, classical MLE is inconsistent while MWLE is consistent.
Limitations
- Relies on regularity assumptions such as positive definite Fisher information and unique minimum of the loss.
- Threshold sample size N may not be tight, especially for logistic regression and phase retrieval.
- Misspecified optimality of MWLE is shown only for a constructed worst-case scenario.
Experimental setup
Metrics: excess risk minimax risk ratio
Baselines: maximum weighted likelihood estimator
Builds on: importance weighting maximum weighted likelihood estimator Cramer-Rao bound
Closest papers by meaning
- Minimax Optimal Two-Stage Algorithm For Moment Estimation Under Covariate Shift (ICLR 2025 · similarity 0.87)
- On the benefits of maximum likelihood estimation for Regression and Forecasting (ICLR 2022 · similarity 0.87)
- Overparameterization Improves Robustness to Covariate Shift in High Dimensions (NeurIPS 2021 · similarity 0.87)
- Adapting to Continuous Covariate Shift via Online Density Ratio Estimation (NeurIPS 2023 · similarity 0.86)
- A Theoretical Analysis on Independence-driven Importance Weighting for Covariate-shift Generalization (ICML 2022 · similarity 0.86)
- When Shift Happens - Confounding Is to Blame (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.