Research map / Reinforcement learning / Policy gradient and actor-critic
Safety-Polarized and Prioritized Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAXSAFE, a chance-constrained bi-level safe RL framework that learns state-action reachability estimates to build optimal action masks, implemented in DQN via safety polarization and safety-prioritized replay (SPOM / SPOM_PER).
Problem
Safe RL where unsafe events are sparse but highly consequential; existing CMDP and action-masking methods use dense costs or uniform state-agnostic thresholds and may not reach minimal unsafe probability.
Approach
Minimize undiscounted unsafe probability first, then maximize return among safest actions; learn a state-action reachability estimation function (SA-REF) via backward induction to define state-dependent action masks, softened into Q-values by a log-based polarization function, and prioritize replay by SA-REF TD errors.
Key results
- SPOM_PER achieves the highest overall SWU score of 0.98 across six tasks, versus 0.74 for SPOM and 0.60 for the best baselines (SafeQ and Recovery).
- SPOM_PER attains SWU 1.22 on Roundabout and 1.07 on ACC, the highest in those tasks.
- Results averaged over 6 random seeds with 95% confidence intervals in training curves.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add safety prioritized experience replay to SPOM (ablation) | SPOM · Overall | SWU score: 0.74 -> 0.98 | Helped |
| Add safety prioritized experience replay to SPOM (ablation) | SPOM · ACC and Circle | crash rate | Helped |
| Use 10 log(x) instead of log(x) as the polarization function (ablation) | SPOM · Intersection and TwoWay | crash rate | Helped |
| Use log(x) instead of 10 log(x) as the polarization function (ablation) | SPOM · Intersection and TwoWay | crash rate | Hurt |
| Use 1 − 1/x instead of 10 log(x) as the polarization function (ablation) | SPOM · Intersection and TwoWay | episode reward | Hurt |
| Apply optimal action masks directly instead of safety polarization (ablation) | SPOM · Intersection | episode reward | Hurt |
| Apply optimal action masks directly instead of safety polarization (ablation) | SPOM · TwoWay | episode reward | Hurt |
| Use SPOM PER instead of SafeQ | DQN-based safe RL · Overall | SWU score: 0.60 -> 0.98 | Helped |
| Use SPOM PER instead of Recovery | DQN-based safe RL · Overall | SWU score: 0.60 -> 0.98 | Helped |
Surprising findings
- Direct optimal action masking (OAM) without polarization is overly conservative early in training because the ψ network is immature, hurting reward.
Limitations
- Theoretical convergence is proven only for tabular MDPs; deep version is heuristic.
- Assumes unsafe states are absorbing and that a zero-unsafe-probability policy space exists.
- PPO-based CMDP baselines failed to optimize due to sparse crash cost, so comparisons rely mainly on DQN-based baselines.
Experimental setup
Datasets: highway-env TwoWay highway-env Merge highway-env Roundabout highway-env Intersection ACC Circle
Benchmarks: highway-env TwoWay highway-env Merge highway-env Roundabout highway-env Intersection ACC Circle
Metrics: SWU score episode reward crash rate
Baselines: SafeQ Recovery RL RCDQN RewsDQN DQN RESPO PPOLag
Builds on: Ganai et al. 2023 RESPO reachability estimation function Srinivasan et al. 2020 SafeQ Prioritized experience replay DQN Double Q-learning
Closest papers by meaning
- Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Safety Representations for Safer Policy Learning (ICLR 2025 · similarity 0.89)
- Probabilistic Shielding for Safe Reinforcement Learning (AAAI 2025 · similarity 0.88)
- Sampling-Based Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- Long-Term Safe Reinforcement Learning with Binary Feedback (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.