Research map / Reinforcement learning / Policy gradient and actor-critic

Towards Safe Policy Learning under Partial Identifiability: A Causal Approach

AAAI 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Safe policy improvement · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives sharp closed-form lower bounds on conditional interventional effects by combining observational data with partial-covariate RCT data, and uses them to learn policies guaranteed to beat a deployed baseline policy.

Problem

Learning personalized treatment policies from non-identifiable data (unobserved confounding in observational studies and partially observed covariates in randomized experiments) without a safety guarantee relative to the deployed baseline policy.

Approach

Derives sharp lower bounds on P_x(y, c1, c2) by combining observational data with marginal interventional distributions over partial covariates (Obs+Exp(C1), Obs+Exp(C2), Exp(C1)+Exp(C2)), and then maximizes the worst-case (lower-bound) policy value minus the baseline performance, deviating from worst-case confounding-robust approaches.

Key results

• On synthetic data, the policy using all four bounds (l1-l4) clearly outperforms the behavior policy and random policy across thresholds.
• On the IST trial, learned policies dominate the behavior policy in mean 6-month survival across thresholds, though improvements among bound variants are limited.
• Theorem 1 combining all data is shown to be a sharp lower bound and to consistently dominate the natural bound and the single-source bounds.

Measured effects reported in the paper

Surprising findings

• Bounds using Obs+Exp(C1)+Exp(C2) were found not informative in the synthetic experiment, yet the full l1-l4 policy still performed best.
• Improvement among different bound variants on IST was limited, with most gains coming from the learned policy versus the behavior policy.

Limitations

• Estimation of bounds requires parametric assumptions on the joint distribution (e.g., Gaussian SysBP, independent Bernoulli Age and Sex), so misspecification may bias bounds.
• Algorithm is oracle-based and does not address statistical challenges of estimating the bounds.
• Improvements on the IST dataset are small and limited to a small region of the covariate space.

Experimental setup

Datasets: Synthetic data generated from SCM (Eq. 28) International Stroke Trial (IST) data

Benchmarks: Synthetic data policy evaluation International Stroke Trial policy evaluation

Metrics: mean outcome across treatment thresholds mean outcome vs fraction treated

Baselines: Random policy Behavior policy (C1, C2) Alg 1 with subsets of bounds (l1; l1,l2; l1,l3)

Builds on: Kallus and Zhou 2018 confounding-robust policy improvement Manski natural bounds Lee and Bareinboim 2020 partial observability

Closest papers by meaning

• Confounding-Robust Policy Improvement (NeurIPS 2018 · similarity 0.90)
• Towards Estimating Bounds on the Effect of Policies under Unobserved Confounding (NeurIPS 2024 · similarity 0.88)
• Meta-Learners for Partially-Identified Treatment Effects Across Multiple Environments (ICML 2024 · similarity 0.86)
• When to Act and When to Ask: Policy Learning With Deferral Under Hidden Confounding (NeurIPS 2024 · similarity 0.86)
• Efficient Policy Evaluation Across Multiple Different Experimental Datasets (NeurIPS 2024 · similarity 0.85)
• Learning Representations of Instruments for Partial Identification of Treatment Effects (ICML 2025 · similarity 0.85)