Research map / Reinforcement learning / Policy gradient and actor-critic
Off-Policy Evaluation under Nonignorable Missing Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that complete-case OPE is biased under nonignorable monotone missingness and proposes an IPW value estimator (parametric or semiparametric dropout propensity via shadow variables) with asymptotic inference, validated on simulations and MIMIC-III sepsis data.
Problem
Off-policy evaluation with logged trajectories subject to monotone dropout; existing OPE estimators are not known to be valid when missingness is nonignorable (MNAR), and value inference is needed.
Approach
Treat monotone dropout as a node depending on the next state and reward, show complete-case linear-sieve value estimation is consistent under MAR but biased under MNAR, and correct it with an inverse-probability-weighted estimating equation whose dropout propensity is identified via a shadow variable (parametric or semiparametric exponential tilting), with asymptotic normality for CIs.
Key results
- Under MNAR with n=1000, T=10, CC has bias -0.614 and ECP 0.820, while IPW (P) has bias -0.023 and ECP 0.940.
- Under MAR, CC and IPW both remain nearly unbiased (e.g., bias -0.029 vs -0.03 at n=1000) with ECP around 0.95.
- On a synthetic MNAR-dropout sepsis dataset, the behavior policy value under CC is 0.504 versus IPW 0.988, with the no-dropout CC reference at 1.075.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply IPW instead of complete-case estimation under MNAR | OPE value estimation · Simulation, n=500, MNAR | bias: -0.598 -> -0.016 (IPW P); -0.598 -> 0.015 (IPW SP) | Helped |
| Apply IPW instead of complete-case estimation under MNAR | OPE value estimation · Simulation, n=1000, MNAR | bias: -0.614 -> -0.023 (IPW P); -0.614 -> 0.003 (IPW SP) | Helped |
| Use IPW rather than complete-case estimation under MAR | OPE value estimation · Simulation, n=500, MAR | bias: -0.028 -> -0.025 | No clear effect |
| Use IPW rather than complete-case estimation under MAR | OPE value estimation · Simulation, n=1000, MAR | bias: -0.029 -> -0.03 | No clear effect |
| Apply IPW to MNAR-dropped synthetic sepsis data | OPE value estimation · Synthetic sepsis, Behavior policy | estimated value: 0.504 -> 0.988 (complete-data CC: 1.075) | Helped |
| Apply IPW to MNAR-dropped synthetic sepsis data | OPE value estimation · Synthetic sepsis, DQN | estimated value: 1.437 -> 2.227 (complete-data CC: 2.160) | Helped |
| Apply IPW to MNAR-dropped synthetic sepsis data | OPE value estimation · Synthetic sepsis, Dueling DQN | estimated value: 1.672 -> 2.450 (complete-data CC: 2.368) | Helped |
| Apply IPW to MNAR-dropped synthetic sepsis data | OPE value estimation · Synthetic sepsis, BCQ | estimated value: 1.403 -> 2.147 (complete-data CC: 2.085) | Helped |
| Use IPW rather than complete-case estimation | OPE value estimation · MIMIC-III sepsis, evaluated policies | estimated value: 2.356 -> 2.377; 4.459 -> 4.542; 4.823 -> 4.925; 5.002 -> 5.115 | Mixed |
Surprising findings
- Complete-case estimation remains valid under MAR, so no adjustment is needed in that case.
- On the real sepsis data, IPW yields higher value estimates than CC for all policies, consistent with early-discharged patients being in better condition.
Limitations
- Identification of MNAR relies on a valid shadow variable that is hard to verify and needs domain knowledge.
- Basis-based linear sieve inference requires the state dimension to be low for valid inference.
- Real-data dropout mechanism is unknown, so the true bias cannot be verified.
Experimental setup
Datasets: MIMIC-III v1.4 sepsis cohort simulated MDP data
Benchmarks: simulated MDP MIMIC-III sepsis
Metrics: bias MSE empirical coverage probability confidence interval width
Baselines: Complete Case (CC) estimator Behavior policy value DQN Dueling DQN BCQ
Builds on: linear sieve estimator for Q-function (Ertefaghi-style sieve OPE) shadow variable identification (Miao et al.) exponential tilting semiparametric missing data model fitted Q evaluation
Closest papers by meaning
- Off-Policy Evaluation with Deficient Support Using Side Information (NeurIPS 2022 · similarity 0.91)
- Benchmarks for Deep Off-Policy Evaluation (ICLR 2021 · similarity 0.88)
- State Relevance for Off-Policy Evaluation (ICML 2021 · similarity 0.88)
- OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators (NeurIPS 2024 · similarity 0.88)
- Off-policy Evaluation Beyond Overlap: Sharp Partial Identification Under Smoothness (ICML 2024 · similarity 0.87)
- Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluation (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.