Research map / Reinforcement learning / Policy gradient and actor-critic
A Provable Approach for End-to-End Safe Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PLS, which trains a constrained decision transformer (CDT) offline and then safely tunes its target reward and cost returns online via Gaussian-process bandit optimization, with high-probability lifetime safety and near-optimality guarantees.
Problem
Safe RL policies are hard to keep safe from learning through deployment: online safe RL violates constraints during learning, and offline safe RL policies often misalign actual returns with target returns (unsafe or overly conservative) under distribution shift.
Approach
Learn a return-conditioned policy offline (CDT) and then treat the two target returns (reward, cost) as the only tunable parameters, modeling their mapping to actual returns with GPs and choosing targets via pessimistic/optimistic confidence bounds so that the constraint holds with high probability.
Key results
- PLS is the only method that satisfies the safety constraint in every task at threshold b=20 across the 9 reported tasks (5 runs each).
- PLS achieves the highest reward among safe agents in most tasks, e.g., Hopper-Velocity reward 0.83 vs. 0.68 for CDT (cost 0.42).
- Theory: target-to-actual return gap decomposes into a GP term, a small bias, and an asymptotically negligible term; safety holds with probability at least 1 - Delta.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PLS with offline CDT policy learning and GP optimization of target returns instead of the offline safe-RL baselines | Safe RL policies · Bullet-Safety-Gym and Safety-Gymnasium tasks, b=20 | normalized safety cost | Helped |
| Use PLS with GP-optimized target returns instead of baseline methods | Safe RL policies · Bullet-Safety-Gym and Safety-Gymnasium tasks, b=20 | normalized reward | Mixed |
Surprising findings
- Baselines such as CDT and CCAC often misalign actual and target returns, with both constraint violations and overly conservative behavior observed.
Limitations
- Safety guarantee is probabilistic and depends on accurate GP models, so a few unsafe deployments may occur.
- Near-optimal target returns do not imply a near-optimal policy.
- Theory relies on strong assumptions (near-deterministic transitions, Lipschitz cost, initial safe seed set).
Experimental setup
Datasets: Bullet-Safety-Gym offline datasets Safety-Gymnasium offline datasets
Benchmarks: Bullet-Safety-Gym Safety-Gymnasium
Metrics: normalized reward return normalized safety cost return
Baselines: BCQ-Lag BEAR-Lag CPQ COptiDICE CDT CCAC
Builds on: constrained decision transformer decision transformer SafeOpt-style safe Bayesian optimization return-conditioned supervised learning
Closest papers by meaning
- Long-Term Safe Reinforcement Learning with Binary Feedback (AAAI 2024 · similarity 0.91)
- Offline Safe Reinforcement Learning Using Trajectory Classification (AAAI 2025 · similarity 0.90)
- Probabilistic Shielding for Safe Reinforcement Learning (AAAI 2025 · similarity 0.90)
- Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation (NeurIPS 2024 · similarity 0.90)
- Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.