Research map / Reinforcement learning / MDPs and planning
Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a new risk-sensitive MDP formulation shown equivalent to soft robust MDPs, derives policy gradient theorem with global convergence for tabular direct parameterization, and proposes offline RFZI for KL-soft RMDPs with sample complexity analysis.
Problem
Connecting risk-sensitive MDPs and robust MDPs, and developing policy gradient and sample-based offline learning with theoretical guarantees, since Markov risk measures are not gradient-dominant.
Approach
Defines a risk-sensitive value function where policy randomness is treated risk-neutrally and risk arises only from transition uncertainty; shows equivalence with soft RMDPs whose penalty is independent of the policy, enabling gradient domination and a Z-function fitted iteration for the KL/entropy case.
Key results
- Exact policy gradient under direct parameterization drives the optimality gap to exactly 0 in the 14-state experiments across tested (alpha, beta) settings.
- Theorem gives sum of squared optimality gaps bounded by 16|A|M^4/(1-gamma)^4, implying O(|A|M^4/((1-gamma)^4 eps^2)) iterations for global convergence.
- RFZI sample complexity bound scales with C^2 (beta^{-1} e^{beta/(1-gamma)})^2 log|F| /((1-gamma)^4 eps^2), with worse concentrability dependence than risk-neutral offline RL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase penalty magnitude β | Exact policy gradient · 14-state cycle with fixed mild transition noise (α=0.01) | risk-averse policy behavior | Helped |
| Increase transition noise α | Exact policy gradient · 14-state cycle with fixed moderate penalty magnitude | risk-averse policy behavior | Helped |
| Use a risk-sensitive policy instead of the risk-neutral policy | Exact policy gradient · 14-state cycle, large perturbation magnitude δ | robustness value | Helped |
| Use the learned risk-sensitive policy instead of the robust baseline policy | Exact policy gradient · 14-state cycle under transition perturbations | robustness value | Mixed |
| Increase penalty magnitude β | Exact policy gradient · 14-state cycle across perturbation magnitudes δ | robustness value | Mixed |
| Increase penalty magnitude β | RFZI · 100-state environment under model uncertainty | robustness value | Helped |
| Increase penalty magnitude β | Practical RFZI · 100-state environment | training stability | Hurt |
Surprising findings
- The Markov risk measure is not gradient-dominant, whereas the proposed risk-sensitive formulation is, sharply contrasting prior results.
Limitations
- Policy gradient result limited to exact gradients, not sample-based.
- RFZI designed for KL-soft problems and likely suited only to small action spaces.
- Worse dependence on distributional shift factor M and exponential dependence on beta/(1-gamma) in RFZI bound.
Experimental setup
Datasets: synthetic cycle-graph MDP (14-state and 100-state environments)
Benchmarks: cycle-graph hitting reward MDP (14-state, 100-state)
Metrics: optimality gap average test reward robustness value
Baselines: risk-neutral optimal policy robust baseline policy (KL-rectangular RMDP) RFQI
Builds on: Markov risk measures robust MDP sup-inf formulation RFQI fitted Q iteration
Closest papers by meaning
- A Single-Loop Robust Policy Gradient Method for Robust Markov Decision Processes (ICML 2024 · similarity 0.91)
- Provable Policy Gradient for Robust Average-Reward MDPs Beyond Rectangularity (ICML 2025 · similarity 0.91)
- Accelerated Policy Gradient for s-rectangular Robust MDPs with Large State Spaces (ICML 2024 · similarity 0.91)
- Online MDP with Prototypes Information: A Robust Adaptive Approach (AAAI 2025 · similarity 0.90)
- Solving Robust Markov Decision Processes: Generic, Reliable, Efficient (AAAI 2025 · similarity 0.90)
- Policy Optimization for Robust Average Reward MDPs (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.