Research map / Reinforcement learning / Policy gradient and actor-critic
Combining Experimental and Historical Data for Policy Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes weighted combinations of an experimental-data-only doubly robust estimator and a historical-control-based estimator for policy evaluation, with MSE-optimal weights estimated non-pessimistically or pessimistically, extended to sequential decision making.
Problem
Estimating the average treatment effect of a target policy versus control from a small two-arm experiment when a larger historical dataset under the control policy exists, where reward shift between datasets biases naive pooling.
Approach
Linearly combine an experimental-only DR estimator and a historical-control DR estimator with a weight minimizing estimated MSE; a pessimistic variant replaces the squared reward-shift estimate with an upper confidence bound, making weight selection robust to moderate reward shifts.
Key results
- In the ridesharing-based sequential simulation, the Pessi estimator's MSE increases only slightly as reward shift grows, while Lasso and NonPessi MSEs increase significantly.
- Pessimistic estimator's 95% confidence intervals achieve nominal coverage while being narrower than EDO's; Lasso shows significant undercoverage.
- Non-pessimistic estimator is close to the SPE estimator when reward shift is small, and the pessimistic estimator performs comparably to EDO in the large-shift regime.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPE, which assumes no reward shift | ATE estimation in the non-dynamic simulation · Small reward shift | MSE | Helped |
| Use the proposed non-pessimistic weighted estimator instead of SPE | ATE estimation in the non-dynamic simulation · Small reward shift | MSE | No clear effect |
| Use the proposed pessimistic weighted estimator | ATE estimation in the non-dynamic simulation · Moderate reward shift | MSE | Helped |
| Use the proposed pessimistic weighted estimator instead of non-pessimistic weighting | ATE estimation in the non-dynamic simulation · Moderate reward shift | MSE | Helped |
| Use the proposed pessimistic weighted estimator | ATE estimation in the non-dynamic simulation · Large reward shift | MSE | Helped |
| Integrate historical data with the proposed pessimistic estimator | ATE estimation in the ridesharing sequential simulation · Ridesharing data | MSE | Helped |
| Use pessimistic rather than non-pessimistic weighting | ATE estimation in the ridesharing sequential simulation · Increasing reward shift | MSE | Helped |
| Use NonPessi instead of Lasso | ATE estimation in the ridesharing sequential simulation · Most tested cases | MSE | Helped |
| Use Lasso with a low tuning parameter | ATE estimation in the ridesharing sequential simulation · Small reward shift | MSE | Helped |
| Increase the Lasso tuning parameter | Lasso ATE estimator · Increasing reward shift | MSE | Mixed |
| Use the proposed methods | ATE estimation in the clinical-data-based simulation · ACTG175 | MSE | Helped |
| Use the pessimistic estimator rather than EDO | Confidence intervals for ATE · 95% confidence interval coverage experiment | confidence interval width | Helped |
Surprising findings
- The non-pessimistic weight tends to be smaller than the oracle weight when the reward shift is moderately large, causing significant bias.
- Empirically the spurious estimation error is considerably smaller than the oracle MSE despite being of the same theoretical order.
Limitations
- Non-pessimistic estimator can suffer large MSE under moderate reward shift.
- Hybrid procedure requires prior knowledge of the reward-shift regime and is unreliable when regime must be estimated.
- Sample-split estimator discards efficiency from full-sample use; theory relies on double robustness or convergence rates for nuisance functions.
Experimental setup
Datasets: ACTG175 clinical trial data (2139 patients) ridesharing company A/A experiment data (40 days)
Benchmarks: simulated non-dynamic switchback setting ridesharing-based sequential simulation ACTG175-based simulation
Metrics: MSE CI coverage probability CI length
Baselines: EDO SPE Lasso-based weighted estimator
Builds on: doubly robust estimation double reinforcement learning semi-parametrically efficient estimator for historical controls pessimism in offline RL
Closest papers by meaning
- Efficient Policy Evaluation Across Multiple Different Experimental Datasets (NeurIPS 2024 · similarity 0.89)
- Policy Learning for Balancing Short-Term and Long-Term Rewards (ICML 2024 · similarity 0.86)
- Doubly Optimal Policy Evaluation for Reinforcement Learning (ICLR 2025 · similarity 0.86)
- Efficient Multi-Policy Evaluation for Reinforcement Learning (AAAI 2025 · similarity 0.85)
- Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes (NeurIPS 2024 · similarity 0.85)
- Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.