Research map / Reinforcement learning / Policy gradient and actor-critic
Safety Representations for Safer Policy Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SRPL, which augments RL agent state with a learned state-conditioned safety representation (a distribution over steps-to-cost, learned by a S2C model from recent experience) to reduce conservatism and constraint violations during safe policy learning.
Problem
Safe RL agents, penalized heavily for early constraint violations, become overly conservative (primacy bias), getting trapped in local optima and under-exploring risky but high-reward regions.
Approach
Learn a state-centric safety representation modelled as a categorical distribution over steps-to-cost from recent agent experience across diverse policies (via a FIFO replay buffer), and concatenate it to the state so policies can reason about proximity to unsafe states without restricting exploration.
Key results
- On AdroitHandPen, SR-CRPO reaches return 8800 ± 985 with 4626 ± 447 failures vs CRPO 7893 ± 2576 and 5752 ± 1146 failures.
- On SafeMetaDrive, SR-CPO reduces total cost to 1997 ± 151 from CPO's 2715 ± 366 while success rate rises from 0.088 to 0.47.
- On PointGoal1, SR-CPO improves return from 6.25 to 11.63 with cost-rate 1.49e-2 vs CPO 1.61e-2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add learned safety representations to CRPO | CRPO · AdroitHandPen | Return: $7893\pm 2576$ -> $\textbf{8800} \pm \textbf{985}$ | Helped |
| Add learned safety representations to CRPO | CRPO · AdroitHandPen | Failures: $5752 \pm 1146$ -> $\textbf{4626} \pm \textbf{447}$ | Helped |
| Add learned safety representations to CRPO | CRPO · SafeMetaDrive | Success rate: $0.22 \pm 0.24$ -> $\textbf{0.53} \pm \textbf{0.18}$ | Helped |
| Add learned safety representations to CPO | CPO · SafeMetaDrive | Total cost: $2715 \pm 366$ -> $\textbf{1997} \pm \textbf{151}$ | Helped |
| Add learned safety representations to CPO | CPO · PointGoal1 | Return: $6.25 \pm 3.28$ -> $\textbf{11.63} \pm \textbf{2.28}$ | Helped |
| Add learned safety representations to SauteRL | SauteRL · PointGoal1 | Cost-rate: $1.08 \pm 0.02$ -> $\textbf{1.05} \pm \textbf{0.01}$ | Helped |
| Add learned safety representations to CRPO | CRPO · PointButton1 | Return: $1.47 \pm 0.68$ -> $\textbf{3.12}\pm \textbf{0.99}$ | Helped |
| Add learned safety representations to SauteRL | SauteRL · PointButton1 | Cost-rate: $1.09 \pm 0.02$ -> $\textbf{1.04} \pm \textbf{0.01}$ | Helped |
| Use expected failure likelihood for the current policy instead of a state-centric steps-to-cost distribution (ablation) | CPO with safety representations · Experiments shown in Alternative Safety Representations | Task performance | No clear effect |
| Train the proximity distribution only on on-policy rollouts (ablation) | CPO with safety representations · Experiments shown in Alternative Safety Representations | Task performance | Hurt |
| Use learned safety representations with RGB observations instead of the CPO observation setup (ablation) | CPO · PointGoal1, RGB | Return: $2.54 \pm 1.838$ -> $8.134 \pm 4.46$ | Helped |
| Use learned safety representations with RGB observations instead of the CPO observation setup (ablation) | CPO · PointGoal1, RGB | Cost-rate: $2.0 \pm 0.334$ -> $1.7 \pm 0.374$ | Helped |
Surprising findings
- Learning the safety representation from a policy-agnostic, recent-experience buffer outperforms a policy-dependent safety representation and a expected-violation representation that already exists in the cost critic.
- Using higher-dimensional observations (depth, RGB) degrades baseline performance, but the safety representation gains become larger in those settings.
Limitations
- Struggles to capture long-horizon causal safety mechanisms because distance-to-unsafe beyond the safety horizon is not represented.
- Binary, non-graded definition of safety; does not model degrees of harm.
- Some baselines (CVPO on AdroitHandPen and Ant) were hard to stabilize; cost-limit had to be changed to 0.1 for some tasks.
Experimental setup
Datasets: Island Navigation AdroitHandPen SafeMetaDrive Safety Gym PointGoal1 Safety Gym PointButton1 MuJoCo Ant MuJoCo Hopper MuJoCo Walker2d
Benchmarks: Island Navigation AdroitHandPen SafeMetaDrive Safety Gym PointGoal1 Safety Gym PointButton1 MuJoCo Ant MuJoCo Hopper MuJoCo Walker2d
Metrics: episodic return total failures/costs during training success rate cost-rate Q-value estimates state visitation
Baselines: CPO TRPO-PID SauteRL CRPO CSC CVPO DQN C51 BootstrapDQN
Builds on: CPO CRPO SauteRL CSC CVPO
Closest papers by meaning
- Towards Safe Reinforcement Learning with a Safety Editor Policy (NeurIPS 2022 · similarity 0.92)
- Feasibility Consistent Representation Learning for Safe Reinforcement Learning (ICML 2024 · similarity 0.91)
- Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty (NeurIPS 2026 · similarity 0.90)
- Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis (AAAI 2024 · similarity 0.90)
- Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Explainably Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.