Research map / Label-efficient and robust learning / Recognition and long-tail
Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a double machine learning estimator of a black-box predictor's target risk under joint covariate shift and selective labels, with identification, efficient influence function, and asymptotic normality, evaluated on semi-synthetic and real eICU data.
Problem
Pre-deployment evaluation of a fixed prediction model when the target population differs in covariates (covariate shift) and source outcome labels are observed only for decision-selected units (selective labels).
Approach
Identifies the target risk under selection on observables plus covariate shift and derives its efficient influence function, yielding a doubly robust DML estimator that jointly corrects for selective labeling (IPW) and covariate transport (density ratio) while using unlabeled target covariates.
Key results
- In the semi-synthetic eICU experiment, the DML estimator is least biased across shifted hospitals 443, 199, and 283, while CS-only is positively biased in all target hospitals.
- DML achieves among the lowest RMSE across hospitals; in hospital 199 SL-only matches DML while CS-only and plug-in have higher error.
- On real eICU outcomes, the source and SL-only estimators significantly underestimate BNP MAE on hospitals 443 and 199 relative to DML.
Surprising findings
- The plug-in estimator, a naive combination of both corrections, shows significant bias and wide intervals despite being approximately unbiased in synthetic experiments.
- SL-only estimator's apparent bias decrease under covariance shift is an artifact of a constant risk estimate.
Limitations
- Identification relies on untestable selection-on-observables and overlap assumptions that may fail with unmeasured severity indicators.
- Bootstrap intervals may be invalid for non-asymptotically-linear baselines; real-world results lack an oracle ground truth.
- Bias increases under unobserved selection and conditional outcome shift in robustness checks.
Experimental setup
Datasets: eICU Collaborative Research Database
Benchmarks: eICU semi-synthetic target risk eICU real-world lab-outcome target risk synthetic robustness checks
Metrics: bias RMSE mean absolute error (target risk)
Baselines: Plug-in Estimator CS-only DML estimator SL-only DML estimator
Builds on: double machine learning prediction-powered inference inverse probability weighting
Closest papers by meaning
- Tracking the risk of a deployed model and detecting harmful distribution shifts (ICLR 2022 · similarity 0.89)
- Quantifying Uncertainty in the Presence of Distribution Shifts (NeurIPS 2025 · similarity 0.85)
- Post-Selection Distributional Model Evaluation (NeurIPS 2026 · similarity 0.84)
- On the Impact of Performative Risk Minimization for Binary Random Variables (ICML 2025 · similarity 0.84)
- Estimating Model Performance Under Covariate Shift Without Labels (NeurIPS 2025 · similarity 0.84)
- Training ML Models with Predictable Failures (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.