Research map / Reinforcement learning / Policy gradient and actor-critic

Spectral-Risk Safe Reinforcement Learning with Convergence Guarantees

NeurIPS 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Safety layers and shielding

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SRCPO, a bilevel method for spectral risk measure-constrained RL that splits the problem into a policy-gradient inner problem with dual weights and a sampler-based outer problem over dual variables, with tabular convergence proofs and Safety Gymnasium and legged-locomotion experiments.

Problem

Risk-constrained RL with spectral risk measure (e.g., CVaR) constraints is hard because the nonlinearity of risk measures prevents policy-gradient methods from guaranteeing convergence to a global optimum.

Approach

Use the dual form of spectral risk measures to turn the constrained problem into a bilevel problem: an inner policy-gradient update with risk advantage functions that are linear in performance difference, and an outer sampler distribution over discretized dual parameters trained with a softmax/cross-entropy update toward the optimal dual variable.

Key results

• Across tasks, SRCPO achieves the highest reward among methods whose cost rates satisfy the thresholds (five seeds).
• WCSAC-Dist attains the highest reward on Safety Gymnasium tasks but violates cost thresholds, attributed to underestimating the risk measure.
• Convergence to an optimal policy and optimal sampler is proven only in the tabular setting.

Measured effects reported in the paper

Surprising findings

• WCSAC-Dist gets the highest reward on Safety Gymnasium tasks yet exceeds the cost thresholds, showing that expectation-of-state-wise-risk approximations can underestimate the constraint.

Limitations

• Convergence guarantees hold only in the tabular setting; function approximation and linear MDPs are not covered.
• Outer problem sampler requires discretizing the spectrum into M parameters, with approximation error bounded by 1/M.
• Practical implementation uses quantile distributional critics and a truncated normal sampler, which departs from the tabular theory.

Experimental setup

Datasets: Safety Gymnasium point/car goal and button tasks legged robot locomotion tasks (bipedal, quadrupedal)

Benchmarks: Safety Gymnasium point goal Safety Gymnasium car goal Safety Gymnasium button tasks legged robot locomotion (quadrupedal, bipedal)

Metrics: reward sum cost rate cost distribution percentiles

Baselines: CVaR-CPO CPPO WCSAC-Dist SDPO SDAC

Builds on: CPO PCPO P3O CVaR-CPO WCSAC

Closest papers by meaning

• Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning (ICML 2025 · similarity 0.87)
• Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents (NeurIPS 2025 · similarity 0.87)
• Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
• Risk-Sensitive Reward-Free Reinforcement Learning with CVaR (ICML 2024 · similarity 0.87)
• Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation (ICML 2024 · similarity 0.87)
• Flipping-based Policy for Chance-Constrained Markov Decision Processes (NeurIPS 2024 · similarity 0.86)