Research map / Reinforcement learning / MDPs and planning

Risk-averse Total-reward MDPs with ERM and EVaR

AAAI 2025 · Reinforcement learning › MDPs and planning · approach: Robust MDP policy gradient · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that the risk-averse total reward criterion under ERM and EVaR admits optimal stationary deterministic policies in transient MDPs with mixed-sign rewards, and gives exponential value iteration, policy iteration, and LP algorithms plus an EVaR reduction to ERM.

Problem

Risk-averse MDPs with monetary risk measures (ERM, EVaR) under discounted criteria require history-dependent policies and lack DP equations; prior total-reward results with exponential utilities restrict rewards to a single sign.

Approach

Prove that the ERM and EVaR total reward criterion in transient MDPs, allowing both positive and negative rewards, is optimized by a stationary deterministic policy satisfying an exponential Bellman equation; use a squeeze argument exploiting transience, and reduce EVaR to a finite set of ERM problems over discretized risk levels.

Key results

• Optimal ERM total-reward policies are stationary deterministic whenever the value is bounded, under transience and mixed-sign rewards.
• EVaR total-reward optimum is approximated within delta by a finite set of ERM-TRC problems solved via linear programming.
• In the gambler's ruin experiment, optimal EVaR policies range from always quitting (alpha=0.2) to bold betting (alpha=0.9).

Measured effects reported in the paper

Surprising findings

• Discounting with the termination probability does not preserve optimal policies under risk aversion, unlike the risk-neutral case.
• Nested CVaR total-reward values can be unbounded even for transient MDPs.

Limitations

• Analysis restricted to finite state and action spaces
• Assumes initial distribution strictly positive for simplicity
• Numerical sensitivity to irrational coefficients left to future work

Experimental setup

Benchmarks: gambler's ruin tabular MDP

Metrics: ERM value EVaR value final capital distribution

Baselines: discounted ERM/EVaR nested CVaR total reward

Builds on: exponential utility MDPs discounted EVaR MDPs transient MDP theory

Closest papers by meaning

• Risk-Averse Total-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.94)
• Online Learning in Risk Sensitive constrained MDP (ICML 2025 · similarity 0.88)
• Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
• Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithm (NeurIPS 2024 · similarity 0.88)
• A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents (ICML 2025 · similarity 0.87)
• Provably Efficient Partially Observable Risk-sensitive Reinforcement Learning with Hindsight Observation (ICML 2024 · similarity 0.87)