Research map / Reinforcement learning / Policy gradient and actor-critic

Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learning

NeurIPS 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Importance-weighted OPE estimators · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a two-way unmeasured confounding assumption (trajectory-specific plus time-specific latent confounders) and a neural tensor network deconfounder that jointly learns latent embeddings, transition dynamics, and behavior policy for model-based off-policy evaluation.

Problem

Off-policy evaluation in sequential decision making when unmeasured confounders affect both actions and transitions/rewards, where standard no-unmeasured-confounding methods and one-way deconfounding models are biased or too restrictive.

Approach

Model unmeasured confounders as the sum of time-invariant trajectory embeddings and trajectory-invariant time embeddings (two-way fixed effects), learn them jointly with a neural tensor network that captures their interaction with observations, and plug them into a Monte Carlo model-based policy value estimator.

Key results

• On the simulated dynamic process (DP, policy A), TWD achieves LMSE 4.23 versus 4.72 for TWD-TO and 6.78-7.50 for one-way variants in the ablation (1000 trajectories).
• On the tumor growth example, TWD achieves LMSE 4.42 (policy A) and 4.33 (policy B), while TWD-TO is best at 3.58 and 3.39.
• On MIMIC-III, TWD and MB are reported to rank the individualized policy highest, while other baselines give conflicting rankings; TWD gives the lowest prediction errors.

Measured effects reported in the paper

Surprising findings

• TWD-TO (transition-only loss) outperforms full TWD on the tumor growth example, contrary to the joint-learning claim.
• The unconstrained unmeasured confounding model overfits, and the one-way model underfits in the linear simulation.

Limitations

• Assumes all confounders are either trajectory-specific or time-specific; confounders that vary across both dimensions or depend on policy are not handled.
• Sensitivity analysis shows performance is very sensitive to violation of the assumption in the tumor growth example.
• Requires an observation that together with the two-way confounders blocks backdoor paths; pre-treatment variables may bias estimates in M-graphs.

Experimental setup

Datasets: simulated dynamic process tumor growth PK-PD simulation MIMIC-III

Benchmarks: simulated dynamic process tumor growth example MIMIC-III

Metrics: logarithmic mean squared error (LMSE) bias mean squared prediction error root MSE

Baselines: Model-based (MB) OPE Minimax weight learning (MWL) Double robust (DR) Partially observable MWL Partially observable DR Recurrent state-space model (RSSM) TWDIDP1 TWDIDP2

Builds on: deconfounder algorithm (Wang & Blei) two-way fixed effects model neural tensor network doubly inhomogeneous decision process

Closest papers by meaning

• Model-Free and Model-Based Policy Evaluation when Causality is Uncertain (ICML 2021 · similarity 0.91)
• Skill or Luck? Return Decomposition via Advantage Functions (ICLR 2024 · similarity 0.87)
• Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluation (ICML 2025 · similarity 0.86)
• Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation (NeurIPS 2024 · similarity 0.86)
• Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPs (ICLR 2025 · similarity 0.86)
• Off-policy Evaluation Beyond Overlap: Sharp Partial Identification Under Smoothness (ICML 2024 · similarity 0.86)