Research map / Reinforcement learning / MDPs and planning
A Policy Gradient Method for Confounded POMDPs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an offline policy gradient method for confounded POMDPs with continuous states/observations, identifying the policy gradient via bridge functions and estimating it with min-max NPIV, then running gradient ascent with global convergence guarantees.
Problem
Learning history-dependent optimal policies in confounded, partially observable MDPs with continuous state and observation spaces from offline data, where unobserved states confound actions, rewards and transitions, so standard off-policy policy gradients are biased.
Approach
Identifies the history-dependent policy gradient (and value) via value and gradient bridge functions defined by sequential conditional moment restrictions using proxy observations O_0 and O_t, estimates them by min-max NPIV with general function classes, and plugs the estimate into gradient ascent with a non-asymptotic suboptimality bound.
Key results
- Theorem gives a policy gradient estimation error of order O_P(1/sqrt(N)) uniformly over the policy class.
- Suboptimality bound decomposes into O(1/sqrt(N) + 1/sqrt(K)) terms plus an approximation error, matching MDP results.
- In simulations, the proposed method reaches the in-class optimal policy while the naive method has irreducible bias and behavioral cloning performs significantly worse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed policy-gradient estimator instead of the naive estimator that treats observations as states | Offline confounded POMDP policy optimization · Simulation study | policy value | Helped |
| Use the proposed policy-gradient method instead of behavioral cloning | Offline confounded POMDP policy optimization · Simulation study | policy value | Helped |
Surprising findings
- A naive policy gradient estimator that ignores the unobserved state stays biased no matter how large the sample size is.
Limitations
- Requires full coverage assumption on the behavior policy, which may fail in settings like electronic medical records.
- Bounds scale polynomially with horizon (e.g., T^9.5 suboptimality constant, L scales with T^4), and history dependence increases complexity.
- Requires completeness and Fisher non-degeneracy conditions, and results are for a finite action space.
Experimental setup
Datasets: simulated offline POMDP data
Benchmarks: simulated confounded POMDP
Metrics: policy value vs iterations
Baselines: naive policy gradient estimator behavioral cloning
Builds on: min-max estimator for conditional moment restrictions proximal causal inference bridge functions policy gradient convergence analysis for MDPs
Closest papers by meaning
- Model-based Reinforcement Learning for Confounded POMDPs (ICML 2024 · similarity 0.92)
- A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processes (ICML 2022 · similarity 0.92)
- Sequential Monte Carlo for Policy Optimization in Continuous POMDPs (NeurIPS 2025 · similarity 0.91)
- Multi-Environment POMDPs with Finite-Horizon Objectives (NeurIPS 2026 · similarity 0.89)
- Scalable Policy-Based RL Algorithms for POMDPs (NeurIPS 2025 · similarity 0.88)
- A Surprisingly Simple Continuous-Action POMDP Solver: Lazy Cross-Entropy Search Over Policy Trees (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.