Research map / Reinforcement learning / MDPs and planning
From POMDP Theory to Deep RL with Particle Filters
Kevin Tan, Ziping Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Connects POMDP theory to practical deep RL by analyzing particle-filter-based belief inference, latent dynamics learning, and planning under belief in partially observable environments.
Problem
Model-based RL in partially observable environments requires inferring hidden states, learning latent dynamics, and planning under the belief; practical methods approximate these parts in poorly understood ways, while POMDP theory analyzes computationally inefficient algorithms.
Approach
Bridges POMDP theory and deep RL by using particle filters as the belief-inference component, with theoretical analysis of the full pipeline (inference, latent dynamics, planning) rather than one aspect in isolation. Details beyond the abstract are unknown.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase particle count J from 8 to 512 (ablation) | PF-MCTQ · Mixing Tiger | belief-approximation error: 0.144 -> 0.012 | Improved |
| increase MCTS rollouts N from 4 to 512 (ablation) | PF-MCTQ · Mixing Tiger | Monte Carlo RMSE: 0.171 -> 0.015 | Improved |
| increase planning depth d from 0 to 12 (reduce QMDP bias) (ablation) | PF-MCTQ · Mixing Tiger | structural error: 0.314 -> 0.0216 | Improved |
| reduce policy update-to-data ratio to one update per ten steps, restore online model updates and weight decay (ablation) | PF-MCTQ · MuJoCo Ant (partial observation) | average return, 40k-49.2k steps: -396 -> 893 (position hidden); -518 -> 822 (velocity hidden) | Improved |
| add GRU backbone instead of memoryless MLP (ablation) | PF-MCTQ · Double-integrator LQR | belief tracking | Improved |
| use PF-MCTQ instead of RLSVI-SMC (tailored linear-Gaussian method) | RLSVI-SMC · HeartSteps | cumulative reward | Hurt |
| use PF-MCTQ instead of recurrent model-free baselines in low-sample regime | Recurrent SAC / GRU-PPO · MuJoCo Hopper and Walker2d (partial observation) | return | Mixed |
| hide position instead of velocity | PF-MCTQ · HalfCheetah (partial observation) | return | Hurt |
Closest papers by meaning
- Scalable Policy-Based RL Algorithms for POMDPs (NeurIPS 2025 · similarity 0.90)
- Model-based Reinforcement Learning for Confounded POMDPs (ICML 2024 · similarity 0.89)
- Rethinking Transformers in Solving POMDPs (ICML 2024 · similarity 0.88)
- Towards General-Purpose Model-Free Reinforcement Learning (ICLR 2025 · similarity 0.88)
- Provable Representation with Efficient Planning for Partially Observable Reinforcement Learning (ICML 2024 · similarity 0.88)
- The Power of Resets in Online Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
