Research map / Reinforcement learning / Policy gradient and actor-critic
Safe Reinforcement Learning using Finite-Horizon Gradient-based Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Gradient-based Estimation (GBE), a first-order gradient method for estimating finite-horizon non-discounted constraints in differentiable Safe RL, and CGPO, a trust-region constrained policy optimizer built on it.
Problem
Existing deep Safe RL methods estimate constraint changes with infinite-horizon discounted advantage functions (ABE), which yields large errors for finite-horizon non-discounted cost constraints and causes safety-violating updates.
Approach
Estimate objective and constraint values for the next policy with a first-order Taylor expansion using analytic gradients back-propagated along finite trajectories in differentiable environments, then solve a trust-region surrogate sub-problem with closed-form dual solutions and an adaptive radius.
Key results
- CGPO converges in 1.77e+06 steps on CartPole versus 2.32e+06 for the best baseline (SHAC-LAG), with 3.96% violation ratio versus 6.59% for APPO.
- On Reacher, CGPO reaches a 14.29% violation ratio versus 30.68% for CPO, and 1.15e+06 convergence steps versus 1.98e+06 for CPO.
- The paper reports ABE relative estimation error exceeding 1.0 while GBE error is much smaller.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CGPO instead of the strongest reported baseline | Safe RL policy optimization · CartPole | Conv. Steps: 2.32e+06 -> 1.77e+06 | Helped |
| Use CGPO instead of the strongest reported baseline | Safe RL policy optimization · CartPole | Vio. Ratio (%): 5.96 -> 3.96 | Helped |
| Use CGPO instead of the strongest reported baseline | Safe RL policy optimization · Reacher | Conv. Steps: 1.98e+06 -> 1.15e+06 | Helped |
| Use CGPO instead of the strongest reported baseline | Safe RL policy optimization · Reacher | Vio. Ratio (%): 46.04 -> 14.29 | Helped |
| Use CGPO instead of the strongest reported baseline | Safe RL policy optimization · HalfCheetah | Conv. Steps: 2.32e+06 -> 2.02e+06 | Helped |
| Use CGPO instead of the strongest reported baseline | Safe RL policy optimization · HalfCheetah | Vio. Ratio (%): 18.33 -> 7.31 | Helped |
| Use CGPO instead of the strongest reported baseline | Safe RL policy optimization · Ant | Conv. Steps: 9.29e+06 -> 8.52e+06 | Helped |
| Use CGPO instead of the strongest reported baseline | Safe RL policy optimization · Ant | Vio. Ratio (%): 9.54 -> 6.44 | Helped |
| Replace ABE with GBE for estimating finite-horizon constraints (ablation) | Safe RL constraint estimation · FunctionEnv and HalfCheetah | relative estimation error | Helped |
| Use an adaptive trust-region radius instead of a static radius (ablation) | CGPO · Safe RL tasks | performance and constraint satisfaction | Helped |
| Add a World Model to provide gradients (ablation) | CGPO in environments with limited differentiability · non-differentiable tasks | performance and safety constraint satisfaction | Helped |
Surprising findings
- The ABE estimator's relative error exceeds 1.0 even on a simple task, indicating near-useless constraint estimates.
- Lagrangian baselines (SHAC-LAG, BPTT-LAG) oscillate around the constraint threshold due to delayed dual variable response.
Limitations
- Relies on environment differentiability; poorly differentiable systems (HalfCheetah) show larger estimation error.
- An excessively small trust region radius can reduce update efficiency.
- World-model extension adds computational overhead and gives low-precision gradients with a two-layer MLP.
Experimental setup
Datasets: Brax CartPole Brax Reacher Brax HalfCheetah Brax Ant
Benchmarks: Brax CartPole Brax Reacher Brax HalfCheetah Brax Ant
Metrics: episodic return constraint violation ratio convergence steps relative estimation error
Baselines: PDO APPO CPO CUP BPTT-Lag SHAC-Lag
Builds on: SHAC BPTT CPO CUP PDO
Closest papers by meaning
- Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.91)
- Constrained Update Projection Approach to Safe Policy Optimization (NeurIPS 2022 · similarity 0.90)
- Sampling-Based Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
- Off-Policy Primal-Dual Safe Reinforcement Learning (ICLR 2024 · similarity 0.89)
- A Provable Approach for End-to-End Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement Learning (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.