Research map / Reinforcement learning / MDPs and planning
Provable Representation with Efficient Planning for Partially Observable Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes μLV-Rep, a multi-step latent variable representation for L-decodable POMDPs that linearly represents Q-functions, enabling tractable least-squares planning and optimism-based exploration, with PAC analysis and Meta-world visual control experiments versus DreamerV2 and MWM.
Problem
Efficient, practical RL in partially observable MDPs: planning, learning and exploration are computationally and statistically intractable for POMDPs, and existing structured-POMDP methods rely on intractable oracles.
Approach
An L-step decodable POMDP admits a linear representation of the Q-function in a latent variable p(z|x_h,a_h) via a moment matching policy, which lets policy evaluation reduce to least squares and makes optimism bonuses computable; learned with a variational ELBO and practical planning via SAC.
Key results
- On four reported Meta-world tasks, μLV-Rep shows faster convergence and higher final success than DreamerV2 and MWM over 1M steps, averaged over 5 seeds.
- Across 50 Meta-world tasks, μLV-Rep achieves better than 90% success on 33 tasks and performs better or within 10% of the best baseline on 41 tasks.
- Training speed of 21.3 steps per second versus 8.1 for MWM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use μLV-Rep instead of MWM | visual robotic manipulation on Meta-World · Meta-World | training speed (steps per second): 8.1 -> 21.3 steps per second | Helped |
| Use μLV-Rep instead of DreamerV2 and MWM | visual robotic manipulation · four reported Meta-World tasks | success rate | Helped |
| Use μLV-Rep instead of the best of DreamerV2 and MWM | visual robotic manipulation · Meta-World 50 tasks | number of tasks with better or within 10% of best baseline performance: 41 out of the 50 tasks | Helped |
Limitations
- Theoretical guarantees rely on finite candidate model class, realizability and kernel regularity assumptions; sample complexity is exponential in L via |A|^L.
- Empirical evaluation limited to Meta-world visual tasks; classic POMDP benchmarks such as RockSample left for future study.
- Practical implementation uses VAE pretraining online and a fixed L=3 history length.
Experimental setup
Datasets: Meta-world
Benchmarks: Meta-world
Metrics: success rate training speed (steps per second)
Baselines: DreamerV2 Masked World Model (MWM)
Builds on: LV-Rep linear MDP DreamerV2 Soft Actor-Critic moment matching policy
Closest papers by meaning
- Partially Observable Hierarchical Reinforcement Learning with AI Planning (Student Abstract) (AAAI 2024 · similarity 0.91)
- Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy (NeurIPS 2024 · similarity 0.91)
- Sequential Monte Carlo for Policy Optimization in Continuous POMDPs (NeurIPS 2025 · similarity 0.90)
- Provable Partially Observable Reinforcement Learning with Privileged Information (NeurIPS 2024 · similarity 0.90)
- Provably Efficient Partially Observable Risk-sensitive Reinforcement Learning with Hindsight Observation (ICML 2024 · similarity 0.89)
- Sample-Efficient Learning of POMDPs with Multiple Observations In Hindsight (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.