Research map / Reinforcement learning / Policy gradient and actor-critic
Efficient Policy Evaluation Across Multiple Different Experimental Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Develops graphical identification criteria and doubly-robust (DML) estimators for policy effects in a target domain by combining multiple experimental source datasets collected under different policies and domain discrepancies.
Problem
Estimating the effect of a candidate (possibly conditional or stochastic) policy in a target domain from finite samples of multiple experimental and observational source datasets that differ in distribution (external validity / transportability).
Approach
Extends selection-diagram-based transportability identification to multi-stage policies across multiple source domains, and derives a multiply-robust DML estimator whose error decomposes into a sum of mixed-bias products so that it converges at n^{-1/2} even when nuisances converge slowly.
Key results
- On synthetic simulations the proposed DML estimator shows faster convergence than OM and PW estimators for both two-experiment and multi-experiment settings.
- On ACTG 175 with noisy nuisances, regression and DML estimators converge faster than PW; DML does not consistently beat OM at all sample sizes.
- On Project STAR, all estimators improve with sample size and DML shows faster convergence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DML instead of OM or PW | Policy evaluation on synthetic data · Synthetic simulations: two and multiple experiments | absolute error | Helped |
| use DML instead of OM or PW | Policy evaluation on ACTG 175 · ACTG 175 | absolute error | Mixed |
| add noise when learning nuisances | OM and DML policy estimators · ACTG 175 | absolute error | Helped |
| increase dataset size | PW, OM, and DML policy estimators · Project STAR | absolute error | Helped |
| use DML instead of PW or OM | Policy evaluation on Project STAR · Project STAR | absolute error | Helped |
Surprising findings
- DML does not consistently outperform the outcome-model estimator on ACTG 175 at all sample sizes, since its error inherits errors from the PW component.
Limitations
- Guarantees are asymptotic/infinite-sample; finite-sample uncertainty not quantified.
- Requires correct causal and selection diagrams from domain knowledge.
- Computational cost of nuisance and density-ratio estimation in high dimensions.
Experimental setup
Datasets: ACTG 175 Project STAR synthetic SCM simulations
Benchmarks: synthetic SCM simulations ACTG 175 (semi-synthetic) Project STAR (semi-synthetic)
Metrics: absolute error
Baselines: OM (outcome model) estimator PW (probability weighting) estimator
Builds on: selection diagrams generalized transportability double machine learning off-policy evaluation
Closest papers by meaning
- Combining Experimental and Historical Data for Policy Evaluation (ICML 2024 · similarity 0.89)
- Towards Estimating Bounds on the Effect of Policies under Unobserved Confounding (NeurIPS 2024 · similarity 0.88)
- OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators (NeurIPS 2024 · similarity 0.86)
- Predictive Performance Comparison of Decision Policies Under Confounding (ICML 2024 · similarity 0.86)
- Towards Safe Policy Learning under Partial Identifiability: A Causal Approach (AAAI 2024 · similarity 0.85)
- Efficient Multi-Policy Evaluation for Reinforcement Learning (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.