Research map / Reinforcement learning / Visual and representation RL
Explainably Safe Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes explainable shielding for safe RL by representing the shield as a hierarchy of decision trees (risk-category tree, action-permission trees, and execution trees for unsafe actions), built from a probabilistic world model and safety specification.
Problem
Shields in safe RL are large lookup tables that are hard for humans to interpret, and naive decision-tree representations of nondeterministic shields are too large to be explainable.
Approach
Hierarchical, top-down case-based explanations: a design-time Level 1 decision tree classifies states into safe/critical/dangerous/unsafe from value-iteration risk; runtime Level 2 trees explain allowed actions in a state's leaf, and Level 3 execution trees show traces that make an action unsafe.
Key results
- On Frozen Lake grids from 5x5 to 50x50, the Level 1 and Level 2 decision trees are several orders of magnitude smaller than the shielding lookup table.
- Hierarchical Level 1 and Level 2 trees are in most Highway scenarios more compact than a single shield tree; Level 1 for HW3-c is slightly under 70 nodes.
- Shield computation takes 0.15 s (5x5) to 15.4 s (50x50) on Frozen Lake; tree learning takes about 2 s per tree.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| represent the shield with hierarchical decision trees instead of only a lookup table | Explainably Safe RL · Frozen Lake | representation size | Helped |
| use hierarchical Level-1 and Level-2 explanations instead of one single tree | Explainably Safe RL · Highway Cruise Control | representation size | Helped |
| train with a safety shield instead of without a shield | PPO · Frozen Lake | task success | Helped |
| train with a safety shield instead of without a shield | PPO · Highway HW3-c | crashes | Mixed |
| increase the proportion of holes from approximately 4% to 20% of the grid | Explainably Safe RL · Frozen Lake | decision tree size | Hurt |
| add a predicate for special Highway HW3-c cases | Explainably Safe RL · Highway HW3-c | Level-1 decision tree size | Helped |
Surprising findings
- Denser hole placement (about 20% of cells) makes decision trees grow because dangerous and critical states become harder to separate.
Limitations
- Relies on user-defined predicates for compact trees; automatic predicate generation is left to future work.
- Explanations depend on an abstract world model; world-model inaccuracy led to some crashes in shielded training on HW3-c.
- Guarantees hold only for the model; no user study on understandability.
Experimental setup
Datasets: Farama Frozen Lake (randomly generated instances) Farama Highway environment (HW2-f, HW2-c, HW3-f, HW3-c) Boeing Taxinet environment
Benchmarks: Frozen Lake Highway cruise control (HW2-f, HW2-c, HW3-f, HW3-c) Boeing Taxinet
Metrics: shield lookup table size decision tree node count computation time
Baselines: MaskablePPO PPO
Builds on: shielded RL (Alshiekh et al.) probabilistic shielding (Jansen et al.) dtControl Tempest
Closest papers by meaning
- Probabilistic Shielding for Safe Reinforcement Learning (AAAI 2025 · similarity 0.90)
- Safety Representations for Safer Policy Learning (ICLR 2025 · similarity 0.90)
- Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis (AAAI 2024 · similarity 0.87)
- AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.