Research map / Reinforcement learning / MDPs and planning
Sequential Monte Carlo for Policy Optimization in Continuous POMDPs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Casts POMDP policy learning as inference in a nested Feynman-Kac model and optimizes history- or belief-dependent policies with a particle-based policy gradient (P3O), outperforming QMDP-based RL baselines on continuous POMDP benchmarks.
Problem
Policy optimization in continuous POMDPs where agents must balance information gathering (exploration) against reward exploitation; existing RL methods often rely on QMDP approximations that ignore the value of future observations.
Approach
Formulate the belief-space objective as a risk-sensitive log-marginal likelihood of optimality events under a non-Markovian Feynman-Kac model, estimate the policy gradient via Fisher's identity using samples from a nested SMC (outer trajectory filter, inner belief filters) that targets the optimal trajectory distribution.
Key results
- On the light-dark and triangulation tasks, P3O and DVRL consistently outperform QMDP-based SLAC and DualSMC, which struggle on tasks requiring deliberate exploration.
- On the tractable LQG benchmark, P3O converges toward the optimal LQG performance without exploiting analytic structure.
- Increasing Feynman-Kac and belief filter particle counts improves approximation of the target and learning performance, with diminishing returns.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use P3O without exploiting analytic LQG structure | P3O · LQG | cumulative reward | Helped |
| use P3O rather than QMDP-based solvers | P3O · Light-Dark | average return | Helped |
| use DVRL rather than QMDP-based solvers | DVRL · Light-Dark | average return | Helped |
| use P3O rather than QMDP-based solvers | P3O · Triangulation | average return | Helped |
| increase the number of Feynman–Kac and belief-filter particles (ablation) | P3O · POMDP benchmarks | learning performance | Helped |
| lower temperature η (ablation) | P3O · POMDP benchmarks | learning convergence | Hurt |
| increase temperature η (ablation) | P3O · LQG | policy-gradient variance | Helped |
| use too high a temperature η (ablation) | P3O · POMDP benchmarks | learning performance | Hurt |
Surprising findings
- Belief-dependent policies learn faster on control tasks but can be constrained by particle belief quality, as bootstrap particle filters degenerate in some cases (cart-pole trade-off).
- Lower temperature eta (approaching REINFORCE) leads to high-variance gradients and slower convergence, while too high eta exacerbates path degeneracy.
Limitations
- Bootstrap particle filter may not scale to high-dimensional settings due to proposal-target mismatch.
- Belief tracker requires oracle access to the observation likelihood.
- Bias of the score estimate is not quantified; tuning the risk parameter eta remains open.
Experimental setup
Benchmarks: LQG pendulum cart-pole light-dark triangulation
Metrics: average return
Baselines: SLAC DVRL DualSMC QMDP
Builds on: Feynman-Kac models nested sequential Monte Carlo control-as-inference soft actor-critic REINFORCE
Closest papers by meaning
- A Policy Gradient Method for Confounded POMDPs (ICLR 2024 · similarity 0.91)
- Provable Representation with Efficient Planning for Partially Observable Reinforcement Learning (ICML 2024 · similarity 0.90)
- Scalable Policy-Based RL Algorithms for POMDPs (NeurIPS 2025 · similarity 0.90)
- A Surprisingly Simple Continuous-Action POMDP Solver: Lazy Cross-Entropy Search Over Policy Trees (AAAI 2024 · similarity 0.89)
- Simplifying Complex Observation Models in Continuous POMDP Planning with Probabilistic Guarantees and Practice (AAAI 2024 · similarity 0.89)
- Multi-Environment POMDPs with Finite-Horizon Objectives (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.