Research map / Reinforcement learning / MDPs and planning
A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes two meta-algorithms (optimistic and policy-gradient) that reduce OCE risk-sensitive RL to risk-neutral RL in an augmented budget MDP, with regret/PAC and global-convergence/monotone-improvement guarantees.
Problem
Risk-sensitive RL with static optimized certainty equivalent (OCE) objectives, where optimal policies are history-dependent and no Bellman equations hold for the static risk.
Approach
Augment the MDP with a known, deterministic budget state and an OCE-based augmented reward so that OCE RL reduces to risk-neutral RL with Markovian augmented policies and an optimized initial budget; then plug in optimistic oracles (UCB-VI, Rep-UCB, GOLF) or policy-optimization oracles (NPG) on top.
Key results
- Optimistic meta-algorithm with UCB-VI attains OCE values above the best Markovian policy for all six tested OCEs, e.g. CVaR_0.25 of 0.75 vs 0.5 for the best Markovian policy in the proof-of-concept MDP.
- Gradient-based meta-algorithm with NPG as oracle reaches the optimal non-Markovian CVaR_0.25 value of 0.75 across three neural oracles (PPO forward/backward KL, REINFORCE).
- Theory gives OCE regret of Otilde(V^max_u sqrt(SAHK)) for tabular MDPs and first risk-sensitive bounds for exogenous block MDPs via GOLF with discrete rewards.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the OCE optimistic meta-algorithm with UCB-VI instead of the best Markovian policy | Synthetic MDP · Synthetic MDP | E[X] − Var(X): 0.95 -> 1.07± 0.01 | Helped |
| Use the OCE optimistic meta-algorithm with UCB-VI instead of the best Markovian policy | Synthetic MDP · Synthetic MDP | E[X] − 2Var(X): 0.5 -> 0.81± 0.01 | Helped |
| Use the OCE optimistic meta-algorithm with UCB-VI instead of the best Markovian policy | Synthetic MDP · Synthetic MDP | Entr−1.0(X): 1.25 -> 1.25± 0.01 | No clear effect |
| Use the OCE optimistic meta-algorithm with UCB-VI instead of the best Markovian policy | Synthetic MDP · Synthetic MDP | Entr−2.0(X): 0.91 -> 0.90± 0.02 | Hurt |
| Use the OCE optimistic meta-algorithm with UCB-VI instead of the best Markovian policy | Synthetic MDP · Synthetic MDP | CVaR0.25(X): 0.5 -> 0.75± 0.02 | Helped |
| Use the OCE optimistic meta-algorithm with UCB-VI instead of the best Markovian policy | Synthetic MDP · Synthetic MDP | CVaR0.5(X): 1.0 -> 1.12± 0.03 | Helped |
Surprising findings
- For entropic risk, where Markovian policies are known to be optimal, the proposed algorithms still learn the optimal policy.
- Neural policy-gradient runs first plateau at a Markovian policy (~2k steps) before converging to the optimal history-dependent policy (~8k steps).
Limitations
- Global convergence and gradient guarantees require a discrete reward assumption (finite support of cumulative reward).
- GOLF-based optimistic oracle is computationally inefficient due to version-space optimism.
- Experiments are limited to a single two-state proof-of-concept MDP.
Experimental setup
Benchmarks: two-state proof-of-concept MDP
Metrics: OCE value CVaR_0.25 value mean-variance value entropic risk value
Baselines: best Markovian policy distributional RL trajectory Q-learning
Builds on: CVaR augmented MDP UCB-VI Rep-UCB GOLF natural policy gradient
Closest papers by meaning
- Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.93)
- Risk-Sensitive Reward-Free Reinforcement Learning with CVaR (ICML 2024 · similarity 0.91)
- Provably Efficient CVaR RL in Low-rank MDPs (ICLR 2024 · similarity 0.91)
- Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents (NeurIPS 2025 · similarity 0.91)
- Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation (ICML 2024 · similarity 0.90)
- Online Learning in Risk Sensitive constrained MDP (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.