Research map / Reinforcement learning / MDPs and planning
Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes sharp, semiparametrically efficient orthogonal estimators for best- and worst-case policy value in infinite-horizon MDPs under a transition-kernel marginal sensitivity model, with robust FQE and minimax w-function estimation as nuisances.
Problem
Off-policy evaluation when the test MDP may differ from the observed MDP (due to confounding, distribution shift, or adversaries); the goal is sharp best/worst-case bounds on policy value from transition data, with valid inference.
Approach
Extends the marginal sensitivity model from single-step causal inference to infinite-horizon MDPs by bounding transition-density ratios, derives the sharp robust Bellman equation via CVaR, and builds a cross-fitted orthogonal estimator from the efficient influence function that is efficient and asymptotically normal even with slow nuisance rates.
Key results
- On a synthetic MDP with Lambda in {1,2,4,8} and 10 replications, the orthogonal estimator (Orth) achieves the lowest MSE for the true worst-case policy value across all Lambda values.
- All three estimators (Q, W, Orth) are consistently valid across the tested Lambda values, per the authors.
- Incorporating robust importance-sampling weights improves the Q-only robust FQE estimator, even though W alone is much noisier.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine robust importance-sampling weights with the RobustFQE estimate | RobustFQE (Q), yielding Orth · Synthetic MDP (Λ=1,2,4,8) | MSE | Helped |
| Use robust importance-sampling weights alone instead of the orthogonal estimator | Robust policy value estimation · Synthetic MDP (Λ=1,2,4,8) | MSE | Hurt |
Surprising findings
- Importance-sampling weights alone (W) are much noisier, yet combining them with Q-estimates still improves over Q alone.
Limitations
- Experiments are limited to a simple one-dimensional synthetic MDP (plus a sepsis application mentioned but not detailed in the text)
- Requires completeness and quantile-regression oracle assumptions for robust FQE and minimax w-learning
- Assumes i.i.d. (s,a)-rectangular confounding and constant or known sensitivity parameter Lambda
Experimental setup
Datasets: synthetic one-dimensional MDP (20,000 tuples per replication) MIMIC-III (sepsis management)
Benchmarks: synthetic robust-OPE MDP
Metrics: mean squared error range of policy value estimates coverage/validity across sensitivity parameter Lambda
Baselines: Q-only robust FQE W-only robust minimax importance sampling
Builds on: double reinforcement learning marginal sensitivity model minimax indirect learning fitted-Q evaluation robust MDPs with s,a-rectangularity
Closest papers by meaning
- Semiparametrically Efficient Off-Policy Evaluation in Linear Markov Decision Processes (ICML 2023 · similarity 0.92)
- Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexity (ICLR 2024 · similarity 0.89)
- Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning (NeurIPS 2024 · similarity 0.88)
- RL in Latent MDPs is Tractable: Online Guarantees via Off-Policy Evaluation (NeurIPS 2024 · similarity 0.88)
- Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPs (ICLR 2025 · similarity 0.88)
- On the Curses of Future and History in Future-dependent Value Functions for Off-policy Evaluation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.