Research map / Reinforcement learning / Policy gradient and actor-critic
Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a risk-aware constrained RL framework that places OCE (CVaR) risk measures on reward values in both objective and constraints, solved via a partial Lagrangian and SGDA wrapper around black-box RL solvers such as PPO.
Problem
Standard constrained RL uses expected returns for objectives and constraints, which ignores risky or catastrophic tail events; the paper targets risk-aware constrained RL with reward-based risk measures.
Approach
Formulate objectives and constraints with reward-based OCE (CVaR) risk measures, reduce to a parameterized partial Lagrangian that is exact under a constraint qualification, and solve it with SGDA over (t, lambda) while using any black-box RL solver (PPO) for the policy.
Key results
- On Safety-Gymnasium Point level-1 tasks, the method reports zero cumulative cost across Button, Circle, Goal and Push, versus PPO costs of 150.76, 206.74, 45.09 and 38.48.
- Rewards under the constrained method were lower in Button, Circle and Goal (e.g., Button 2.58 vs 24.29 for PPO) but higher in Push (2.42 vs 0.93).
- The converged CVaR threshold t matches the beta-upper quantile of the post-training velocity distribution.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CVaR-constrained PPO instead of unconstrained risk-neutral PPO | PPO · Button | evaluation cost: 150.76 -> 0.0 | Helped |
| Use CVaR-constrained PPO instead of unconstrained risk-neutral PPO | PPO · Circle | evaluation cost: 206.74 -> 0.0 | Helped |
| Use CVaR-constrained PPO instead of unconstrained risk-neutral PPO | PPO · Goal | evaluation cost: 45.09 -> 0.0 | Helped |
| Use CVaR-constrained PPO instead of unconstrained risk-neutral PPO | PPO · Push | evaluation cost: 38.48 -> 0.0 | Helped |
| Use CVaR-constrained PPO instead of unconstrained risk-neutral PPO | PPO · Button | evaluation reward: 24.29 -> 2.58 | Hurt |
| Use CVaR-constrained PPO instead of unconstrained risk-neutral PPO | PPO · Circle | evaluation reward: 60.18 -> 39.19 | Hurt |
| Use CVaR-constrained PPO instead of unconstrained risk-neutral PPO | PPO · Goal | evaluation reward: 21.89 -> 13.56 | Hurt |
| Use CVaR-constrained PPO instead of unconstrained risk-neutral PPO | PPO · Push | evaluation reward: 0.93 -> 2.42 | Helped |
Surprising findings
- Constraint satisfaction yielded higher reward than the unconstrained PPO agent in the Push environment.
Limitations
- Full strong duality holds only under a constraint qualification; unconditional exactness is open.
- Higher computational cost since each lambda/t update requires approximately solving for a policy.
- Requires extended training and careful step-size tuning for lambda and t.
Experimental setup
Datasets: Safety-Gymnasium (Button, Circle, Goal, Push, velocity tasks) MuJoCo-v4 locomotion
Benchmarks: Safety-Gymnasium Button Safety-Gymnasium Circle Safety-Gymnasium Goal Safety-Gymnasium Push MuJoCo locomotion safe velocity
Metrics: cumulative episodic cost episodic reward CVaR of velocity distribution
Baselines: PPO
Builds on: PPO reward-based risk-averse RL (RCVaR) convex analytic RL formulation primal-dual constrained RL
Closest papers by meaning
- A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents (ICML 2025 · similarity 0.91)
- Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.90)
- Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learning (NeurIPS 2023 · similarity 0.90)
- Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively (AAAI 2024 · similarity 0.90)
- Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.89)
- Constrained Reinforcement Learning Under Model Mismatch (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.