Research map / Reinforcement learning / Policy gradient and actor-critic

Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning

ICLR 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Safety layers and shielding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LSPC, which models safe behavior with a CVAE in latent space and trains a reward-AWR latent encoder to maximize reward under inferred safety constraints for safe offline RL.

Problem

Safe offline RL requires maximizing reward while strictly satisfying cost constraints using only a static dataset, where overly restrictive constraints hurt reward and overly loose ones allow unsafe out-of-distribution actions.

Approach

Learn a CVAE over state-action pairs weighted by cost advantage to form a conservatively safe latent policy, then train a state-conditioned latent encoder with reward-advantage weighted regression whose outputs are squashed by tanh into a restricted high-probability latent region, yielding LSPC-O.

Key results

• LSPC-O reaches average normalized reward 0.72 with cost 0.29 on MetaDrive, versus FISOR 0.36/0.08 and CDT 0.42/0.80.
• LSPC-O averages reward 0.40 with cost 0.27 on Safety Gym and reward 0.54 with cost 0.20 on Bullet Safety Gym, the best safe reward among methods on Bullet Safety Gym.
• Theoretical bounds on performance gap and constraint violation scale with (1/(1-gamma)^2)(sqrt(eps1'/2)+sqrt(eps2'/2)); sample complexity analysis gives decay rate O(1/N^(0.25+v)).

Measured effects reported in the paper

Surprising findings

• Reported that BC-Safe, which uses only safe data subsets, often outperforms FISOR and CDT in reward despite lacking reward-label use.

Limitations

• Choosing the latent restriction epsilon (degree of restriction) is not theoretically linked to cost value and must be tuned.
• Sample complexity analysis is worst-case and relies on Donsker-type assumptions on the policy class.
• Some Bullet Safety Gym tasks use a different restriction value (0.6) than others, indicating per-domain tuning.

Experimental setup

Datasets: DSRL benchmark (MetaDrive, Safety Gymnasium, Bullet Safety Gym offline datasets)

Benchmarks: DSRL MetaDrive DSRL Safety Gymnasium DSRL Bullet Safety Gym

Metrics: normalized reward return normalized cost return

Baselines: BC-Safe CDT CPQ FISOR

Builds on: Implicit Q-learning advantage-weighted regression CPQ FISOR DSRL

Closest papers by meaning

• Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model (ICLR 2024 · similarity 0.90)
• Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL (NeurIPS 2025 · similarity 0.90)
• Offline Safe Reinforcement Learning Using Trajectory Classification (AAAI 2025 · similarity 0.90)
• Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning (AAAI 2025 · similarity 0.90)
• Safety Representations for Safer Policy Learning (ICLR 2025 · similarity 0.90)