Research map / Reinforcement learning / Policy gradient and actor-critic
Embedding Safety into RL: A New Take on Trust Region Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C-TRPO, a trust-region policy optimization method whose divergence adds a barrier-like cost term so trust regions stay inside the safe policy set, with a C-NPG variant and convergence/safety analysis.
Problem
Safe reinforcement learning in CMDPs: existing methods either violate constraints during training (CPO oscillates around the constraint) or sacrifice reward (projection, penalty, log-barrier, Lagrangian methods).
Approach
Replace the state-averaged KL trust-region divergence with a Bregman divergence induced by a mirror function that adds barrier terms phi(b - c^T d), so trust regions contain only safe policies; a surrogate using the cost advantage makes it implementable in TRPO style, with a recovery step and hysteresis.
Key results
- C-TRPO achieves reward competitive with leading safe RL baselines across 8 Safety Gymnasium tasks while being safe on the last iterate, unlike CPO and CUP.
- C-TRPO reaches notably lower cost regret than high-return baselines such as TRPO-Lag, FOCOPS, and CUP, comparable to PCPO but not as low as P3O.
- Theory: C-NPG flow keeps the safe parameter set invariant and converges to the optimal safe policy; C-TRPO approaches CPO as beta goes to 0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use C-TRPO instead of CPO or CUP | Safe policy optimization on Safety Gymnasium tasks · Safety Gymnasium (8 tasks) | final constraint violation | Helped |
| use C-TRPO instead of high-return safe policy optimization algorithms | Safe policy optimization on Safety Gymnasium tasks · Safety Gymnasium (8 tasks) | cost regret | Helped |
| use C-TRPO instead of benchmark algorithms | Safe policy optimization on Safety Gymnasium tasks · Safety Gymnasium (8 tasks) | expected reward | No clear effect |
| use C-TRPO instead of PCPO | Safe policy optimization on Safety Gymnasium tasks · Safety Gymnasium (8 tasks) | cost regret | No clear effect |
| use C-TRPO instead of P3O | Safe policy optimization on Safety Gymnasium tasks · Safety Gymnasium (8 tasks) | cost regret | Hurt |
| increase the safety parameter β above 1 | C-TRPO · Safety Gymnasium tasks | expected reward | Hurt |
| use β ≤ 1 | C-TRPO · Most environments | expected return | No clear effect |
| add the modified trust region to the method (ablation) | C-TRPO · Safety Gymnasium tasks | safety violations | Helped |
| add hysteresis-based recovery (ablation) | C-TRPO · Safety Gymnasium tasks | safety violations | Helped |
Surprising findings
- Low-cost-regret behavior of P3O comes at the price of reward because its final iterate sits with a wide margin from the threshold.
Limitations
- Estimating the proposed divergence is difficult and finite-sample properties are not analyzed.
- CMDPs constrain average cost, so trajectory-wise or state-wise safety constraints are harder to model.
- Performance depends on accuracy of cost advantage and value estimates; higher beta degrades return due to noisy divergence estimates.
Experimental setup
Datasets: Safety Gymnasium
Benchmarks: Safety Gymnasium (4 navigation, 4 locomotion tasks)
Metrics: final cost normalized reward cost regret
Baselines: CPO PCPO CPPO-PID PPO-Lag TRPO-Lag FOCOPS CUP IPO P3O
Builds on: TRPO Constrained Policy Optimization (CPO) Natural Policy Gradient GAE Safety Gymnasium
Closest papers by meaning
- Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (ICLR 2018 · similarity 0.90)
- Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraints (NeurIPS 2023 · similarity 0.90)
- Feasible Policy Optimization for Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
- Scalable Constrained Policy Optimization for Safe Multi-agent Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively (AAAI 2024 · similarity 0.89)
- Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.