Research map / Reinforcement learning / Policy gradient and actor-critic

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning

ICML 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Distributional value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes QR-SRM, a quantile-based distributional RL algorithm that optimizes static spectral risk measures via a closed-form outer update, and uses return distributions to interpret intermediate risk preferences.

Problem

Risk-sensitive RL with fixed per-step risk measures yields overly conservative, time-inconsistent policies, and prior static-risk methods were limited to CVaR; the interpretability of learned risk-sensitive policies is also unclear.

Approach

Optimize a static spectral risk measure (convex combination of CVaRs) by alternating a closed-form outer update of the concave function h (from the current policy's return quantiles) with a distributional Q-learning inner loop on an state-augmented MDP (accumulated discounted reward and discount factor); the coherent-risk decomposition is used to recover intermediate risk preferences for interpret

Key results

• On mean-reversion trading, QR-SRM(phi_alpha=0.5) attains CVaR_0.5 0.27 vs 0.28 for QR-CVaR and 0.27 for QR-SRM with phi_nu=2.0 range, while QR-DQN scores -0.24 on CVaR_0.5.
• On Windy Lunar Lander, QR-SRM with a mixed CVaR spectrum achieves CVaR_0.5 of 14.17 and WSCVaR of 14.57, versus 3.79 and 4.23 for QR-DQN.
• QR-SRM(phi_alpha=1.0) matches QR-DQN closely on Lunar Lander (27.36 vs 32.73 expected return, within one std) despite state augmentation.

Measured effects reported in the paper

Surprising findings

• QR-iCVaR (risk-sensitive action selection with a fixed CVaR per step) underperforms QR-SRM at matched risk levels, supporting the claim that fixed per-step risk measures are suboptimal.
• QR-CVaR showed poor performance in 3 of 5 seeds on Windy Lunar Lander, with unusually low scores.

Limitations

• Value-based method restricted to discrete action spaces; extension to actor-critic needed for continuous actions.
• The outer update provides only a lower bound on the objective; empirically the algorithm converges to policies similar to QR-CVaR, lacking stronger optimality guarantees for h.
• State augmentation enlarges the state space, possibly causing slight performance drop on Lunar Lander.

Experimental setup

Datasets: American put option GBM simulation mean-reversion trading simulation Windy Lunar Lander

Benchmarks: American put option trading mean-reversion trading Windy Lunar Lander

Metrics: expected discounted return CVaR at several levels ERM DPRM weighted sum of CVaRs

Baselines: QR-DQN QR-CVaR QR-iCVaR

Builds on: QR-DQN CVaR state augmentation (Bäuerle & Ott / Miller & Yang) decomposition of coherent risk measures QR-CVaR

Closest papers by meaning

• Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation (ICML 2024 · similarity 0.90)
• Risk-Conditioned Reinforcement Learning: A Generalized Approach for Adapting to Varying Risk Measures (AAAI 2024 · similarity 0.89)
• Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
• A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents (ICML 2025 · similarity 0.88)
• Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
• Provably Efficient CVaR RL in Low-rank MDPs (ICLR 2024 · similarity 0.87)