Research map / Reinforcement learning / MDPs and planning
Predictive Performance Comparison of Decision Policies Under Confounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a partial-identification method for comparing the predictive performance (e.g., TPR/FPR, NPV, expected utility) of a proposed decision policy against a confounded status quo policy, using a delta-regret interval that cancels shared uncertainty and is compatible with MSM, IV, proximal, and Gamma assumptions.
Problem
Comparing predictive performance of a proposed decision policy against an under-specified, confounded status quo policy using observational data where outcomes are only observed for positively decided instances (selective labels).
Approach
Decompose policy performance into identified and partially identified v-statistics and cancel the common partially identified terms in the regret (delta-regret interval), yielding tighter regret bounds than differencing separately computed policy bounds; the framework accepts pointwise bounding functions implied by multiple causal assumptions.
Key results
- The delta-regret interval gives tighter bounds than the baseline regret interval across five performance measures apart from PPV in synthetic experiments under MSM and IV.
- In the healthcare enrollment application, the delta-regret interval for NPV excludes zero across all subpopulations except White patients under 65, while the baseline interval contains zero except for patients aged 65 and over.
- Theorem gives a lower bound on interval improvement proportional to the uncertainty in the cancelled term alpha = overline v_0(0,0) - underline v_0(0,0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the δ-regret interval instead of the baseline regret interval | policy comparisons under confounding · Synthetic data (MSM and IV) | regret interval width | Helped |
| use the δ-regret interval instead of the baseline regret interval | policy comparisons under confounding · Synthetic data (MSM and IV) | positive predictive value regret interval width | No clear effect |
| use the baseline regret interval instead of the δ-regret interval | policy comparisons under confounding · Synthetic data, small-sample settings | coverage of oracle regret | Helped |
| use the δ-regret interval instead of the baseline regret interval | healthcare enrollment policy comparison · Healthcare enrollment, all reported subpopulations except White patients under 65 | NPV regret interval conclusion | Helped |
| use the δ-regret interval instead of the baseline regret interval | healthcare enrollment policy comparison · Healthcare enrollment, patient populations other than those aged 65 and over | NPV regret interval conclusion | Helped |
| use the δ-regret interval instead of the baseline regret interval | healthcare enrollment policy comparison · Healthcare enrollment, high false-negative-cost setting | expected cost regret interval conclusion | Helped |
Surprising findings
- The baseline interval yields better coverage of the oracle regret in small-sample settings because it is more conservative.
- The delta-regret interval gives no improvement over the baseline for PPV in synthetic experiments.
Limitations
- Plug-in estimator converges at slow nonparametric rates; the doubly robust variant is needed for fast rates.
- Real-world application uses a synthetic version of the dataset and narrow analyses, so findings are not conclusive.
- Without additional causal assumptions the regret interval tends to be uninformative; assumption-tailored bounding functions are required.
Experimental setup
Datasets: synthetic data (MSM and IV simulations) synthetic version of healthcare high-risk care management enrollment dataset (~48,000 records)
Benchmarks: synthetic MSM and IV simulations healthcare enrollment policy comparison
Metrics: regret interval width coverage of oracle regret bootstrap confidence intervals
Baselines: baseline regret interval
Builds on: marginal sensitivity model Rosenbaum's Gamma sensitivity model proximal causal inference instrumental variable identification
Closest papers by meaning
- Towards Estimating Bounds on the Effect of Policies under Unobserved Confounding (NeurIPS 2024 · similarity 0.86)
- Efficient Policy Evaluation Across Multiple Different Experimental Datasets (NeurIPS 2024 · similarity 0.86)
- Policy Learning for Balancing Short-Term and Long-Term Rewards (ICML 2024 · similarity 0.85)
- The Relative Value of Prediction in Algorithmic Decision Making (ICML 2024 · similarity 0.85)
- Off-policy estimation with adaptively collected data: the power of online learning (NeurIPS 2024 · similarity 0.85)
- Towards Safe Policy Learning under Partial Identifiability: A Causal Approach (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.