Research map / Reinforcement learning / Policy gradient and actor-critic

Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Safety layers and shielding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Dyna-SAuR, a model-based RL method that jointly learns a control policy and a scalable safety filter (hyperplane-parameterized action filter) from an uncertainty-aware probabilistic ensemble model, evaluated on goal-reaching CartPole and MuJoCo Walker.

Problem

Safe exploration during RL training in high-dimensional systems with unknown dynamics, where classical viability-based safety filters need substantial domain knowledge and scale poorly.

Approach

Train a safety filter entirely in an uncertainty-aware learned model: the filter policy outputs a hyperplane action in the unit hyperball, bijectively mapped to a discriminating hyperplane intersecting the action box, and is trained with RL (FastTD3-based) on a certain-safe-set reward, with data shaping of start states and pink-noise occupancy approximation; the model is retrained as environment d

Key results

• Dyna-SAuR reduces accumulated failures by at least two orders of magnitude compared to the baselines on CartPole and Walker.
• On goal-reaching CartPole, Dyna-SAuR shows the strongest control performance among safe RL approaches and accumulates at most one failure.
• On Walker, Dyna-SAuR performs on par with PPO-Lagrangian in return while being far safer; Infoprop-Dyna gives the highest return but is unaware of safety.

Surprising findings

• Infoprop-Dyna, which ignores safety, consistently outperforms the safe RL baselines in control return.

Limitations

• Efficiently expanding the certain set in high-dimensional spaces remains a challenge; Walker progress is stagnant.
• Requires prior data and a failure set as domain knowledge.
• Exploration in high-dimensional spaces is limited, per the authors.

Experimental setup

Datasets: prior data from LQR controller (CartPole) prior data from partially trained RL agent (Walker)

Benchmarks: goal-reaching CartPole (constrained) MuJoCo Walker (constrained)

Metrics: control return accumulated failures during training

Baselines: PPO-Lagrangian DH-RL Infoprop-Dyna

Builds on: Infoprop-Dyna DH-RL FastTD3 PPO-Lagrangian probabilistic ensemble

Closest papers by meaning

• Sampling-Based Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
• Safety Representations for Safer Policy Learning (ICLR 2025 · similarity 0.90)
• Conservative Safety Critics for Exploration (ICLR 2021 · similarity 0.90)
• Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis (AAAI 2024 · similarity 0.89)
• Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systems (NeurIPS 2024 · similarity 0.88)
• Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation (AAAI 2024 · similarity 0.87)