Research map / Reinforcement learning / Policy gradient and actor-critic

Risk-Averse Online POMDP Planning via CVaR of the Immediate Cost with Performance Guarantees

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Tree search and policy optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Applies CVaR to the per-step immediate cost over the particle belief (rather than to the return) so any expectation-based POMDP planner becomes risk-sensitive by changing only the cost, with finite-time estimation and POMDP–PB-MDP approximation guarantees.

Problem

Risk-averse online POMDP planning: expected cumulative cost can mask dangerous states in the belief tail, and existing CVaR methods (static or iterated/dynamic) target trajectory-level risk and require bespoke algorithms.

Approach

Replace the expected immediate cost under the belief with CVaR of the state-dependent cost at each step, keeping the expected cumulative return as objective so the problem is a standard MDP; the CVaR cost is deterministic given the particle belief, so its error cancels in the Q-error decomposition, giving risk-level-independent Hoeffding bounds.

Key results

• CVaRCost-POMCPOW reduces CVaR cost return on ContinuousLaserTag from 188.0 to 5.4 (97%) and danger-area entries from 4.03 to 0.04.
• CVaRCost-PFT-DPW on LightDark achieves CVaR cost return 144.8 vs 777.7 for PFT-DPW and 250.5 for ICVaR-PFT-DPW.
• On PushPOMDP, CVaR cost variants reduce collisions (1.66-1.78 vs 2.79-2.83) but goal rate drops to 27-33% vs 59-71% for baselines.

Surprising findings

• On PushPOMDP the CVaR cost planners substantially reduce goal completion rate (27-33% vs 59-71%), a safety-task trade-off.
• ICVaR-POMCPOW outperforms CVaRCost-POMCPOW on LightDark (58.6 vs 87.9) and ContinuousLaserTag (8.1 vs 5.4 is reversed; LightDark is the exception).

Limitations

• Per-step CVaR may underestimate trajectory-level tail risk when costs are temporally correlated.
• Approximation bound requires N_p = Omega((d_max/alpha)^2), which is conservative relative to N_p=200 used.
• Analysis covers only PFT-DPW-style and POMCPOW planners; MCTS with adaptive sampling not formally analyzed.

Experimental setup

Benchmarks: LightDark PushPOMDP ContinuousLaserTag PacMan

Metrics: CVaR cost return dangerous encounters goal rate expected return static CVaR

Baselines: PFT-DPW POMCPOW ICVaR-PFT-DPW ICVaR-POMCPOW

Builds on: PFT-DPW POMCPOW particle belief MDP Iterated CVaR (ICVaR)

Closest papers by meaning

• Multi-Environment POMDPs: Discrete Model Uncertainty Under Partial Observability (NeurIPS 2025 · similarity 0.87)
• Simplifying Complex Observation Models in Continuous POMDP Planning with Probabilistic Guarantees and Practice (AAAI 2024 · similarity 0.87)
• Online Learning in Risk Sensitive constrained MDP (ICML 2025 · similarity 0.87)
• Sequential Monte Carlo for Policy Optimization in Continuous POMDPs (NeurIPS 2025 · similarity 0.86)
• A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents (ICML 2025 · similarity 0.86)
• Learning Logic Specifications for Policy Guidance in POMDPs: an Inductive Logic Programming Approach (AAAI 2025 · similarity 0.86)