Research map / Reinforcement learning / MDPs and planning
Probabilistic Shielding for Safe Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a scalable probabilistic shield for safe RL that state-augments a known-safety-dynamics MDP with safety levels computed via sound value iteration, then trains PPO in the shield for strict safety at training and test time.
Problem
Safe RL with undiscounted probabilistic state-avoidance constraints (reach unsafe states with probability at most p) where strict guarantees from linear programming methods scale poorly.
Approach
Augment each MDP state with a real-valued safety level and construct a shield MDP whose actions are constrained so that predicted next safety levels are coherent and above an inductive epsilon-upper bound on unsafe-reach probability; any RL algorithm trained in this shield is guaranteed safe and optimality-preserving as epsilon shrinks.
Key results
- PPO-shield guaranteed safety throughout training and test time in every environment, with reward converging to about 1 in Colour bomb v1/v2 and Bridge crossing v1/v2.
- PPO-shield converged within at most 100,000 steps across all case studies, but converged slower than unshielded PPO, especially on Bridge Crossing.
- PPO-shield significantly outperformed CPO and PPO-Lagrangian in all case studies, which often failed to optimize reward under restrictive safety constraints.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply PPO-Shield instead of unshielded PPO | PPO · All environments | safety constraint violations | Helped |
| apply PPO-Shield instead of unshielded PPO | PPO · Colour bomb gridworld v1 | cumulative reward | No clear effect |
| apply PPO-Shield instead of unshielded PPO | PPO · Colour bomb gridworld v2 | cumulative reward | No clear effect |
| apply PPO-Shield instead of unshielded PPO | PPO · Bridge crossing v1 | cumulative reward | No clear effect |
| apply PPO-Shield instead of unshielded PPO | PPO · Bridge crossing v2 | cumulative reward | No clear effect |
| apply PPO-Shield instead of unshielded PPO | PPO · Media streaming | cumulative reward | Helped |
| apply PPO-Shield instead of CPO and PPO-Lagrangian | Safe RL baselines · All case studies | cumulative reward | Helped |
| use PPO-Shield instead of PPO | PPO · All environments except Bridge crossing v1 and v2 | rate of convergence | Hurt |
| use PPO-Shield instead of PPO | PPO · Bridge crossing v1 and v2 | rate of convergence | Hurt |
| use PPO-Shield | PPO-Shield · All case studies | training steps to converge: 100 000 steps | Helped |
| use PPO-Shield instead of PPO-Lagrangian and CPO | Safe RL baselines · Bridge crossing v1 and v2, Colour bomb v2, Media streaming v2 | convergence: a million steps | Helped |
| use CPO instead of PPO-Lagrangian | Safe RL baselines · Media streaming and Colour bomb gridworld v1, safety bound 0.5 | convergence to optimal policy: 0.5 | Helped |
Surprising findings
- PPO-Lagrangian and CPO failed to converge within a million steps in several environments when the safety bound was relaxed to 0.5 in additional experiments.
Limitations
- Assumes the safety dynamics of the MDP are known.
- Shield state and action spaces are larger than the original MDP, which may slow convergence.
- Constraint violations may occur while computing the upper bound beta unless a safe policy is known in advance.
Experimental setup
Benchmarks: Media streaming Colour bomb gridworld v1 Colour bomb gridworld v2 Bridge crossing v1 Bridge crossing v2 Pacman
Metrics: cumulative reward safety (probability of reaching unsafe states)
Baselines: PPO PPO-Lagrangian CPO
Builds on: PPO Interval Iteration shielding (Alshiekh et al.) Omnisafe
Closest papers by meaning
- Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis (AAAI 2024 · similarity 0.93)
- Long-Term Safe Reinforcement Learning with Binary Feedback (AAAI 2024 · similarity 0.91)
- Sampling-Based Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
- Explainably Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- A Provable Approach for End-to-End Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Flipping-based Policy for Chance-Constrained Markov Decision Processes (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.