Research map / Reinforcement learning / Policy gradient and actor-critic
Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RCRL, a risk-aware Bayesian RL architecture that uses Dirichlet-Categorical transition beliefs to estimate the mean and variance of multi-step risk and gates action selection so the risk stays below a confidence-bounded threshold during training.
Problem
Safe exploration: keeping safety-constraint violations bounded during RL training (not just for the final policy) in MDPs with unknown transition probabilities and limited observability.
Approach
Maintain Dirichlet-Categorical posteriors over transition probabilities, back-propagate expected m-step risk with a first-order Delta-method variance approximation, and use a Cantelli bound to restrict actions to those whose risk is below Phi_max with confidence C(n); a "safety mode" picks the expected-safest action when no action qualifies.
Key results
- On BridgeCross (Prior 1, Phi_max=0.33), RCRL averaged 404.3 successes and 54.2 failures over 500 episodes, versus 414.6 successes and 990.5 failures for Q-learning with penalty over 1500 episodes.
- On Pacman, the risk-aware agent with horizon m=3 exceeded a 75% success rate after 275 episodes with 68 failures, while Q-learning with penalty achieved 0 successes in 1500 episodes.
- With an informative prior (Prior 3) and Phi_max=0.0033, the BridgeCross agent entered unsafe states only 1.1 times on average over 500 episodes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the maximum acceptable risk from Φ_max=0.01 to Φ_max=0.33 with Prior 1 | RCRL · BridgeCross, |A|=5 | successes: 506.0 -> 404.3 | Mixed |
| Increase the maximum acceptable risk from Φ_max=0.01 to Φ_max=0.33 with Prior 1 | RCRL · BridgeCross, |A|=5 | failures: 417.9 -> 54.2 | Helped |
| Use a highly informative Prior 3 instead of uninformative Prior 1 at Φ_max=0.01 | RCRL · BridgeCross, |A|=5 | successes: 506.0 -> 407.4 | Mixed |
| Use Prior 3 instead of Prior 2 at Φ_max=0.01 | RCRL · BridgeCross, |A|=5 | failures: 0.5 -> 14.4 | Hurt |
| Lower Φ_max from 0.01 to 0.0033 with Prior 3 | RCRL · BridgeCross, |A|=5 | failures: 14.4 -> 1.1 | Helped |
| Increase the action space from 5 to 9 actions | RCRL · BridgeCross | successes: The paper states that augmenting the action space does not necessarily lead to an improvement. | Mixed |
| Increase the risk horizon from m=2 to m=3 | RCRL · Pacman | failures: 77 -> 68 | Helped |
| Use RCRL instead of QL with Penalty | Q-learning · Pacman | successes: 0 -> 234 | Helped |
| Use RCRL with Prior 3 and Φ_max=0.0033 instead of QL with Penalty | Q-learning · BridgeCross, |A|=5 | failures: 990.5 -> 1.1 | Helped |
Surprising findings
- An uninformative prior with a very strict Phi_max=0.01 caused degenerate behaviour, with the agent repeatedly selecting the same action in a state and stalling.
- The most informative prior (Prior 3) entered unsafe states more often than Prior 2 at Phi_max=0.01, because it assigned lower risk to moving near hazards.
Limitations
- Variance and mean are approximations via first-order Delta method; the theory is asymptotic and the Cantelli bound is approximate.
- Experiments are limited to two small gridworld tasks with a single run configuration (10 averaged runs for BridgeCross, one agent for Pacman).
- Very strict risk thresholds can prevent exploration of optimal actions, trading off progress for safety.
Experimental setup
Benchmarks: Slippery BridgeCross gridworld (20x20) Pacman gridworld
Metrics: number of successes number of failures (unsafe-state entries) steps to cross per successful episode episodes to 75% success rate
Baselines: Q-learning with penalty
Builds on: Q-learning Dirichlet-Categorical conjugate model Delta method Cantelli inequality
Closest papers by meaning
- Sampling-Based Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.93)
- Probabilistic Shielding for Safe Reinforcement Learning (AAAI 2025 · similarity 0.93)
- Safe Exploration via Policy Priors (ICLR 2026 · similarity 0.92)
- Conservative Safety Critics for Exploration (ICLR 2021 · similarity 0.92)
- Long-Term Safe Reinforcement Learning with Binary Feedback (AAAI 2024 · similarity 0.90)
- Safety Representations for Safer Policy Learning (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.