Research map / Reinforcement learning / MDPs and planning
Long-Term Safe Reinforcement Learning with Binary Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoBiSaRL, a safe RL method for CMDPs with unknown stochastic transitions and binary safety feedback that guarantees long-term (episode-level) safety with high probability by combining GLM confidence bounds with Lipschitz-based pessimistic bounds and a conservative reference policy.
Problem
Safe RL with binary (not numeric) safety feedback, unknown stochastic transitions, and a requirement that safety holds at every step from the current time to the horizon with high probability, without assuming a known safe policy for all states.
Approach
Models the binary safety function with a GLM and derives pessimistic lower bounds on future safety by combining GLM confidence intervals with a Lipschitz bound on how far the state can drift under a conservative policy; a Lagrangian multiplier trades off reward against divergence from that policy when selecting actions from a safe action set.
Key results
- LoBiSaRL incurred 0.0 ± 0.0 unsafe actions versus 3.3 ± 2.2 for the Instantaneous baseline and 26.8 ± 13.6 for the Unsafe agent.
- LoBiSaRL achieved normalized reward 0.76 ± 0.12, below Instantaneous (0.86 ± 0.10) and the Unsafe agent (1.00).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LoBiSaRL's long-term safety constraint instead of the Instantaneous baseline's instantaneous constraint | Synthetic grid-world safe RL · Synthetic grid-world | Unsafe actions: 3.3 \pm 2.2 -> 0.0 \pm 0.0 | Helped |
| Use LoBiSaRL's long-term safety constraint instead of the Instantaneous baseline's instantaneous constraint | Synthetic grid-world safe RL · Synthetic grid-world | Reward: 0.86 \pm 0.10 -> 0.76 \pm 0.12 | Hurt |
Surprising findings
- The Instantaneous baseline sometimes violates the safety constraint due to environment stochasticity, while the proposed method is often overly conservative in reward.
Limitations
- Performance in reward is lower than the instantaneous-constraint baseline; the method is conservative.
- Relies on strong assumptions: Lipschitz feature mapping and a known Lipschitz conservative policy with bounded state drift.
- Evaluated only on a small synthetic grid-world.
Experimental setup
Datasets: synthetic 20x20 grid-world with randomly generated reward and safety functions
Benchmarks: synthetic 20x20 grid-world (100 random environments)
Metrics: normalized reward number of unsafe actions
Baselines: Random Unsafe Linear Instantaneous
Builds on: GLM-based safe exploration (SafeLOG-style logistic bandits) Lipschitz MDP constrained MDP
Closest papers by meaning
- Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.91)
- Probabilistic Shielding for Safe Reinforcement Learning (AAAI 2025 · similarity 0.91)
- Sampling-Based Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
- A Provable Approach for End-to-End Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis (AAAI 2024 · similarity 0.90)
- Flipping-based Policy for Chance-Constrained Markov Decision Processes (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.