Research map / Reinforcement learning / MDPs and planning

Planning and Learning in Average Risk-aware MDPs

NeurIPS 2025 · Reinforcement learning › MDPs and planning · approach: Robust MDP policy gradient · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Extends risk-neutral average-cost MDP planning and Q-learning to average risk-aware MDPs with general dynamic (Markov) risk maps, via a risk-aware relative value iteration, an MLMC-based Q-learning, and an off-policy asynchronous Q-learning for UBSR.

Problem

Average-cost MDPs for continuing tasks become hard to solve when the agent is risk-sensitive; existing model-free average risk-aware Q-learning exists only for the entropic risk measure, and no general RVI or model-free method covers general dynamic risk measures.

Approach

Replace the expectation in relative value iteration and relative Q-iteration with a Markovian risk map; make model-free Q-learning unbiased via multi-level Monte Carlo for general risk maps, and use the root-characterization of UBSR (E[ell(v - SR(v))]=0) to get an off-policy asynchronous update without resampling.

Key results

• Risk-aware RVI and MLMC Q-learning converge to the true optimal average risk on a random 10-state, 5-action MDP, with RVI converging much faster than MLMC Q-learning.
• Off-policy UBSR (expectile) Q-learning converges faster and with lower variance than MLMC Q-learning at an equal sample budget (300,000 asynchronous vs 6,000 synchronous iterations).
• Policies optimal under each risk measure attain their own optimal average risk; e.g. on machine replacement, expectile-optimal policy gives 68.75 vs 54.42 for risk-neutral policy under expectile evaluation.

Measured effects reported in the paper

Surprising findings

• The off-policy UBSR Q-learning algorithm, which lacks a convergence proof, converges faster with lower variance than the provably convergent MLMC Q-learning in the experiments.
• Mean-CVaR satisfies the convergence conditions while plain CVaR does not.

Limitations

• Almost sure convergence of the off-policy UBSR Q-learning algorithm is not proven.
• MLMC Q-learning lacks finite-sample guarantees and requires repeated samples per state-action pair; r<=1/2 implies infinite average samples.
• Requires strong ergodicity (irreducibility and common reachable state) and Lipschitz/bounded-slope conditions on loss functions; experiments limited to small finite MDPs.

Experimental setup

Benchmarks: machine replacement water reservoir management inventory management

Metrics: optimal average risk convergence of average risk estimate

Baselines: value iteration MLMC Q-learning risk-neutral RVI

Builds on: relative value iteration multi-level Monte Carlo Q-learning for robust average MDPs Ruszczynski risk-averse MDP framework utility-based shortfall risk

Closest papers by meaning

• Risk-Averse Total-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexity (ICLR 2024 · similarity 0.89)
• Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
• A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents (ICML 2025 · similarity 0.87)
• Regret Analysis of Policy Gradient Algorithm for Infinite Horizon Average Reward Markov Decision Processes (AAAI 2024 · similarity 0.87)
• Risk-averse Total-reward MDPs with ERM and EVaR (AAAI 2025 · similarity 0.87)