Research map / Reinforcement learning / MDPs and planning

Model-based Reinforcement Learning for Confounded POMDPs

ICML 2024 · Reinforcement learning › MDPs and planning · approach: POMDP policy optimization

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a model-based offline RL method for confounded POMDPs with continuous states/observations, using kernel-based two-stage bridge function estimation and pessimistic policy optimization, with a finite-sample suboptimality bound under partial coverage (theory only, no experiments).

Problem

Offline policy learning in confounded POMDPs with continuous state and observation spaces, where behavior policy depends on unobserved states; prior model-based work was tabular and prior policy-gradient work required full coverage.

Approach

Model-based policy value identification via reward-emission and dynamic-emission bridge functions solving conditional moment restrictions, estimated by a kernel two-stage procedure (with a new risk functional for E[b(W,y)|X]=p(y|X) across all y), followed by pessimistic optimization over confidence regions that do not depend on the policy.

Key results

• Finite-sample suboptimality bound of order N^{-γ/(4γ+4)} (up to constants and log factors) under partial coverage (only concentrability of the optimal policy needed).
• Policy-class size does not enter the bound, unlike the model-free Lu et al. (2022) bound which has a sqrt(log|Π|) term.

Limitations

• No empirical evaluation; practical algorithm is left for future work.
• Assumes realizability and no approximation error for bridge function, density, and conditional mean classes.
• Sequential integration of bridge functions for reward marginals may be computationally expensive.

Experimental setup

Metrics: suboptimality bound (theoretical)

Baselines: Lu et al. 2022 Hong et al. 2023 Tennenholtz et al. 2020

Builds on: Lu et al. 2022 (model-free pessimistic confounded POMDP) Singh et al. 2019 (kernel proxy variable) Mastouri et al. 2021 Tennenholtz et al. 2020 (tabular model-based identification) Hong et al. 2023 (policy gradient confounded POMDP)

Closest papers by meaning

• Provably Efficient Offline Reinforcement Learning for Partially Observable Markov Decision Processes (ICML 2022 · similarity 0.93)
• A Policy Gradient Method for Confounded POMDPs (ICLR 2024 · similarity 0.92)
• Off-Policy Evaluation in Partially Observable Environments (AAAI 2020 · similarity 0.92)
• Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPs (ICLR 2025 · similarity 0.92)
• Scalable Policy-Based RL Algorithms for POMDPs (NeurIPS 2025 · similarity 0.90)
• Optimistic Model Rollouts for Pessimistic Offline Policy Optimization (AAAI 2024 · similarity 0.90)