Research map / Reinforcement learning / Policy gradient and actor-critic

Sampling-Based Safe Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Safety layers and shielding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SBSRL, a model-based safe RL method that enforces expected-cost constraints jointly over a finite set of GP/ensemble dynamics samples and uses an epistemic-uncertainty constraint for exploration, with high-probability safety and finite-time near-optimality guarantees.

Problem

Safe exploration in continuous-state episodic CMDPs with unknown dynamics: the policy at every episode must satisfy the expected cost constraint under the true dynamics while the agent still learns efficiently.

Approach

Replace the intractable worst-case over the GP confidence set with a finite set of prior GP samples truncated to confidence regions, enforce tightened cost constraints jointly on all samples, and drive exploration by constraining the expected epistemic uncertainty along trajectories (inactive when the greedy policy is already informative).

Key results

• On PendulumSwingUp and CartpoleSwingUp (five seeds), SBSRL maintains safety throughout training while reaching near-optimal return, whereas a mean-dynamics constraint baseline violates safety.
• On PendulumSwingUp, SBSRL converges faster to an optimal policy than ActSafe variants with 3 or 5 pure-exploration episodes.
• On a 60 Hz remote-controlled race car, SBSRL initialized offline improves on the prior policy and maintains safety at all times, while the mean-only constraint incurs higher collision costs.

Surprising findings

• Increasing the initial exploration threshold d_sigma^0 can improve convergence in practice despite theory favoring small values.

Limitations

• Theory restricted to finite-horizon episodic CMDPs requiring manual resets
• Worst-case sample count M from small-ball bounds can be large, though fewer samples suffice in practice
• Safety guarantees rely on Gaussian noise, RKHS-bounded prior mismatch, and policy classes such as MPC; deep variant lacks formal guarantees

Experimental setup

Datasets: DeepMind Control Suite PendulumSwingUp DeepMind Control Suite CartpoleSwingUp SafetyGym RWRL offline real-world race car trajectories

Benchmarks: PendulumSwingUp CartpoleSwingUp SafetyGym RWRL real-world race car

Metrics: normalized cumulative return maximum cost violation expected cost kinetic energy lost in collisions

Baselines: ActSafe mean-dynamics constraint baseline

Builds on: ActSafe scenario optimization GP truncated sampling (Lederer et al.) SafeMPC / GP-based safe MBRL

Closest papers by meaning

• Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis (AAAI 2024 · similarity 0.93)
• Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation (NeurIPS 2024 · similarity 0.92)
• Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.92)
• Long-Term Safe Reinforcement Learning with Binary Feedback (AAAI 2024 · similarity 0.91)
• Probabilistic Shielding for Safe Reinforcement Learning (AAAI 2025 · similarity 0.91)
• Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty (NeurIPS 2026 · similarity 0.90)