Research map / Learning theory and algorithms / Prediction and decision losses
Achievable distributional robustness when the robust risk is only partially identified
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the worst-case robust risk and its minimax quantity for partially identifiable distributional robustness in a linear additive-shift SCM, showing existing invariance/anchor methods are suboptimal and validating on synthetic and gene-expression data.
Problem
Distributional robustness methods assume the robust risk is identifiable from training distributions, but in practice it is often only partially identifiable, so existing invariance-based methods lack guarantees and the best achievable robustness is unclear.
Approach
Define the worst-case robust risk as the supremum of the robust risk over all parameters observationally equivalent to the training data, which is always well-defined; its minimizer (worst-case robust predictor) and minimax value give the best achievable robustness. Derived in closed form for a linear additive-shift model with unseen shift directions.
Key results
- In the partially identifiable setting, anchor regression and OLS test error grows with unseen-shift strength gamma' with steeper slopes than the worst-case robust predictor, which closely matches the theoretical lower bound.
- On gene expression data, anchor regression and DRIG are best at 0% unseen directions but become inferior to OLS as the proportion of unseen test shifts grows, while the worst-case robust predictor's test loss stays relatively stable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the worst-case robust predictor instead of anchor regression, DRIG, ICP, or OLS | Single-cell gene expression prediction · Gene expression test sets with unseen shift directions | MSE | Helped |
| Increase the proportion of unseen test shift directions | Anchor regression and DRIG · Gene expression test sets | MSE | Hurt |
| Increase test shift strength | Worst-case robust predictor · Gene expression test sets | MSE | No clear effect |
| Increase test shift strength | Previous invariant methods · Gene expression test sets | MSE | Hurt |
| Evaluate only shifts in training directions | Anchor regression and DRIG · Gene expression test sets with 0% unseen directions | MSE | Helped |
| Add previously unobserved test shift directions | Empirical risk minimization and invariance-based methods · Synthetic Gaussian data, partially identifiable setting | worst-case robust risk (MSE) | Hurt |
| Minimize the empirical worst-case robust loss instead of using OLS or anchor regression | Synthetic Gaussian data · Partially identifiable setting | worst-case robust risk (MSE) | Helped |
| Increase shift strength in training directions | OLS · Synthetic Gaussian data, identifiable setting | worst-case robust risk (MSE) | Hurt |
| Increase shift strength in training directions | Anchor regression · Synthetic Gaussian data, identifiable setting | worst-case robust risk (MSE) | No clear effect |
| Increase strength of unseen shifts | Anchor regression and OLS · Synthetic Gaussian data, partially identifiable setting | worst-case robust risk (MSE) | Hurt |
Surprising findings
- Existing invariance-based robustness methods (anchor regression, DRIG) can perform no better than OLS/ERM once test shifts include unseen directions.
Limitations
- Theory relies on a linear additive-shift setting; extension to nonlinear models and classification is open.
- Requires a bounded causal parameter assumption (norm bounded by C) and a known bound C in practice.
- Worst-case robust predictor is only provably optimal in the linear setting; gene expression results are for a more general shift setting.
Experimental setup
Datasets: single-cell gene expression data (622 genes, observational and interventional environments; 28 genes used) synthetic Gaussian data
Benchmarks: synthetic Gaussian linear SCM experiments single-cell gene expression prediction (28 prediction problems)
Metrics: worst-case robust risk (MSE) test mean squared error
Baselines: OLS / pooled ERM anchor regression DRIG invariant causal prediction (ICP)
Builds on: anchor regression distributionally robust invariant gradients (DRIG) invariant causal prediction instrumental variable regression
Closest papers by meaning
- Bayesian Nonparametrics Meets Data-Driven Distributionally Robust Optimization (NeurIPS 2024 · similarity 0.86)
- Structure-informed Risk Minimization for Robust Ensemble Learning (ICML 2025 · similarity 0.86)
- Distributionally Robust Policy Learning under Concept Drifts (ICML 2025 · similarity 0.86)
- Statistical Properties of Robust Satisficing (ICML 2024 · similarity 0.85)
- The Price of Implicit Bias in Adversarially Robust Generalization (NeurIPS 2024 · similarity 0.85)
- Generalization and Robustness of the Tilted Empirical Risk (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.