Research map / Reinforcement learning / Policy gradient and actor-critic
Externally Valid Policy Evaluation from Randomized Trials Using Additional Observational Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a nonparametric, finite-sample certified method for evaluating decision policies' out-of-sample loss in a target population using RCT data plus target covariates, robust to miscalibrated sampling models bounded by an odds factor Γ.
Problem
External validity of policy evaluation from randomized trials: trial populations differ from the target population, so trial-based loss distributions may not transfer; sampling models are unknown and possibly miscalibrated.
Approach
Bounds the selection odds between target and trial populations by a sensitivity factor Γ around a nominal sampling model, then constructs weighted conformal-style limit curves on the out-of-sample loss using sample splitting, giving finite-sample validity for any Γ-level miscalibration rather than only expected loss via IPSW.
Key results
- On synthetic target population B, the Γ=1 limit curve and benchmark yield negative miscoverage gaps (invalid), while Γ=2 gives positive gaps (valid).
- Informativeness remains above 90% for Γ in [1,2] on synthetic populations; for NHANES, a low-consumption policy certifies with 90% probability that mercury stays below 8 µg/L for Γ in [1,2].
- Benchmarking with omitted covariates suggests Γ in the range 1.5 to 2 is credible for the NHANES application.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the odds-miscalibration allowance from Γ = 1 to Γ = 2 | Proposed certified limit method · Synthetic target population B, treat all policy | miscoverage validity | Helped |
| Use XGBoost instead of logistic regression for the sampling model | Sampling model for policy evaluation · Synthetic target populations B, C, and D | sampling-odds calibration | Helped |
| Use the expected-loss IPSW comparison instead of evaluating tail losses with limit curves | Treat-none and treat-all policies · Synthetic target population A | out-of-sample tail loss: 1.64; 1.54 | Mixed |
Surprising findings
- Expected-loss IPSW comparisons can rank policies differently from tail-loss limit curves: on synthetic population A, treat-none has lower certified tail losses despite a higher expected loss (1.64 vs 1.54).
Limitations
- Requires independent samples; may not hold during outbreaks or similar situations.
- Informativeness degrades as Γ grows and with weak trial-target overlap.
- Γ is unknown and must be chosen via benchmarking or sensitivity heuristics.
Experimental setup
Datasets: NHANES 2013-2014 synthetic Gaussian data
Benchmarks: synthetic target populations A-D NHANES blood mercury
Metrics: informativeness miscoverage gap limit curve tightness
Baselines: IPSW quantile limit (Colnet et al. 2024 / Huang et al. 2021) RCT-only limit curve
Builds on: Ek et al. (2023) Tibshirani et al. (2019) weighted conformal prediction Jin et al. (2023) Huang (2024) benchmarking Huang et al. (2021)
Closest papers by meaning
- Efficient Policy Evaluation Across Multiple Different Experimental Datasets (NeurIPS 2024 · similarity 0.83)
- Prediction-powered Generalization of Causal Inferences (ICML 2024 · similarity 0.83)
- Predictive Performance Comparison of Decision Policies Under Confounding (ICML 2024 · similarity 0.83)
- Towards Estimating Bounds on the Effect of Policies under Unobserved Confounding (NeurIPS 2024 · similarity 0.83)
- Towards Safe Policy Learning under Partial Identifiability: A Causal Approach (AAAI 2024 · similarity 0.83)
- Adaptive Data-Borrowing for Improving Treatment Effect Estimation using External Controls (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.