Research map / Reinforcement learning / MDPs and planning

Risk-Averse Total-Reward Reinforcement Learning

NeurIPS 2025 · Reinforcement learning › MDPs and planning · approach: Robust MDP policy gradient

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes model-free Q-learning algorithms for risk-averse total-reward MDPs under entropic risk measure and EVaR objectives, proving convergence via elicitability and monotonicity of the ERM Bellman operator.

Problem

Model-free computation of optimal stationary policies for risk-averse undiscounted total-reward MDPs with ERM and EVaR objectives, where the Bellman operator is not a contraction and existing methods need known transition probabilities.

Approach

Uses elicitability of the entropic risk measure to rewrite the ERM Bellman operator as a fixed point of an exponential-loss stochastic gradient update, enabling Q-learning with monotonicity-based convergence analysis plus bounded TD-residual conditions; EVaR is handled by reducing it to a finite set of ERM problems over discretized risk levels.

Key results

• On cliff walking, the EVaR optimal policy at alpha=0.2 yields mean return 1.92 versus -0.074 at alpha=0.6 over 48,000 simulated episodes.
• The gap between Q-learning EVaR value and LP-computed EVaR value decreases to zero at roughly 20,000 samples on cliff walking and gambler's ruin.
• EVaR values across six random seeds show a standard deviation of 0.015 at alpha=0.2, indicating stable convergence.

Measured effects reported in the paper

Surprising findings

• Q-learning can converge to different optimal EVaR policies depending on learning rate and number of samples.

Limitations

• Analysis restricted to the tabular setting; no function approximation guarantees.
• Requires choosing z_min and z_max bounds heuristically, which affects detection of unbounded returns.
• Choice of beta_0 requires prior bounds on returns, estimated by extra Q-learning runs.

Experimental setup

Datasets: cliff walking (tabular) gambler's ruin (tabular)

Benchmarks: cliff walking gambler's ruin

Metrics: EVaR value difference from LP-computed EVaR value mean and standard deviation of returns policy optimality

Baselines: linear programming EVaR solution

Builds on: Q-learning elicitability of risk measures stochastic approximation framework of Tsitsiklis / Abounadi et al. EVaR policy computation of Hau et al.

Closest papers by meaning

• Risk-averse Total-reward MDPs with ERM and EVaR (AAAI 2025 · similarity 0.94)
• Planning and Learning in Average Risk-aware MDPs (NeurIPS 2025 · similarity 0.89)
• Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
• Regret Analysis of Policy Gradient Algorithm for Infinite Horizon Average Reward Markov Decision Processes (AAAI 2024 · similarity 0.87)
• Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithm (NeurIPS 2024 · similarity 0.87)
• Online Learning in Risk Sensitive constrained MDP (ICML 2025 · similarity 0.87)