Research map / Reinforcement learning / Policy gradient and actor-critic
Extreme Value Policy Optimization for Safe Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EVO, a constrained RL method that uses extreme value theory (GPD fitted to cost peaks) to define an extreme quantile cost constraint, prioritizes extreme samples in replay, and uses off-policy importance resampling to stabilize tail fitting, reporting near-zero constraint violations on Safety Gymnasium and Safety MuJoCo.
Problem
Expectation-based cost constraints in constrained RL ignore rare, high-impact extreme cost events in the tail, leading to frequent constraint violations even when the expected cost is satisfied.
Approach
Model the tail of the cumulative cost distribution with a Generalized Pareto Distribution from extreme value theory to form an extreme quantile constraint, prioritize replay by GPD quantile levels of reward and cost, and augment the sparse extreme set with importance-resampled off-policy samples.
Key results
- EVO rapidly reaches the feasible domain and keeps zero constraint violations during training in Safety Gym navigation tasks, with return comparable to CPO while CPO violates the threshold on SafetyPointGoal1.
- In Safety MuJoCo, EVO maintains zero violations while WCSAC shows significant violations on SafetyCarCircle1 and SafetyHalfCheetahVelocity.
- On the proposed ratio metric (return divided by violation rate plus constant), EVO achieves the highest score on 7 of 8 environments (e.g., 0.275 vs 0.175 for CPO on SafetyPointGoal1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EVO instead of Simmer | EVO · SafetyCarGoal1 | ratio: 0.212 vs 0.267 | Helped |
| use EVO instead of Saute | EVO · SafetyCarCircle1 | ratio: 0.229 vs 0.237 | Helped |
| use EVO instead of CPO | EVO · SafetyPointGoal1 | ratio: 0.175 vs 0.275 | Helped |
| use EVO instead of WCSAC | EVO · SafetyPointCircle1 | ratio: 0.211 vs 0.236 | Helped |
| use EVO instead of Saute | EVO · SafetyAntVelocity | ratio: 0.245 vs 0.250 | Helped |
| use EVO instead of Saute | EVO · SafetyHalfCheetahVelocity | ratio: 0.231 vs 0.233 | Helped |
| use EVO instead of Simmer | EVO · SafetyHumanoidVelocity | ratio: 0.214 vs 0.219 | Helped |
| use EVO instead of Saute | EVO · SafetySwimmerVelocity | ratio: 0.214 vs 0.213 | Hurt |
| add the EVT-based constraint objective instead of a constant quantile constraint objective (ablation) | EVO · Safety Gymnasium and Safety MuJoCo | policy performance | Helped |
| add extreme prioritization during experience replay (ablation) | EVO · Safety Gymnasium and Safety MuJoCo | policy performance | Helped |
| add off-policy samples for GPD fitting (ablation) | EVO · GPD fitting | variance | Helped |
Surprising findings
- CPO, the expectation-based baseline, violates the constraint threshold on SafetyPointGoal1, so its higher return is not a better policy.
- In the Safety Gym navigation tasks, the initial policy is infeasible, yet EVO converges to the feasible region and then maintains zero violations.
Limitations
- GPD fitting can be inaccurate when the gap between extreme and normal values is subtle, since EVT requires a sufficiently large threshold.
- The theoretical constraint-violation bounds rely on approximations (first/second-order expansions, trust region) and assumptions like the GPD fit holding asymptotically.
- Evaluated only on Safety Gymnasium and Safety MuJoCo simulated tasks with a fixed cost threshold of 25 in main experiments.
Experimental setup
Datasets: Safety Gymnasium (SafetyCarGoal1, SafetyCarCircle1, SafetyPointGoal1, SafetyPointCircle1) Safety MuJoCo (SafetyAntVelocity, SafetyHalfCheetahVelocity, SafetyHumanoidVelocity, SafetySwimmerVelocity)
Benchmarks: Safety Gymnasium Safety MuJoCo
Metrics: average return constraint violation rate constraint value vs. threshold 25 ratio of return to violation rate Kolmogorov-Smirnov fit statistic
Baselines: CPO WCSAC Saute Simmer PPO-Lagrangian RCPO QCPO
Builds on: CPO WCSAC QCPO EVAC PPO
Closest papers by meaning
- Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learning (NeurIPS 2023 · similarity 0.89)
- Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Exploration (ICML 2025 · similarity 0.88)
- Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively (AAAI 2024 · similarity 0.88)
- Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation (NeurIPS 2024 · similarity 0.88)
- Constrained Reinforcement Learning Under Model Mismatch (ICML 2024 · similarity 0.87)
- Safe Reinforcement Learning using Finite-Horizon Gradient-based Estimation (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.