Research map / Reinforcement learning / Policy gradient and actor-critic

Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning

NeurIPS 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Safe policy improvement · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Dynamic Model Predictive Shielding (DMPS), which replaces MPS's task-oblivious backup policy with an MCTS-based local planner that uses the learned Q-function, and trains the neural policy inside this loop to get safe, high-return policies with provable safety during and after training.

Problem

Model predictive shielding guarantees safety in continuous RL but its task-oblivious backup policy halts or takes overly cautious actions, hindering learning and task progress (high recovery regret).

Approach

Replace the fixed backup policy with a dynamic backup policy: a finite-horizon MCTS planner that only expands recoverable states and maximizes n-step discounted reward plus the learned Q-function at the horizon end; the neural policy is trained on the planner's recovery actions. Theory shows recovery regret decays as O(gamma^n) with planning horizon n.

Key results

• DMPS invokes the shield on average 24.7 times per episode across dynamic benchmarks versus 124.1 for MPS.
• DMPS achieves the highest final return in most benchmarks, e.g., 13.0 vs -0.9 for MPS on double-gates+ (DI).
• Theory: recovery regret is bounded by O(gamma^n) (with probability 1 as sampling limit grows), with n the planning horizon.

Measured effects reported in the paper

Surprising findings

• Standard MPS shield invocations increase with training in many scenarios because the learned policy becomes more deterministic and gets blocked by obstacles.
• CPO, PPO-Lag and TD3 fail to achieve positive returns in dynamic benchmarks because they learn to avoid both obstacles and the goal.

Limitations

• Planning compute can grow exponentially with the planning horizon.
• Requires a recoverability check using the backup policy and a model of dynamics.
• REVEL is only evaluated on static benchmarks because it needs a white-box worst-case dynamics model.

Experimental setup

Datasets: obstacle obstacle2 mount-car road road2d dynamic-obst single-gate double-gates double-gates+

Benchmarks: obstacle obstacle2 mount-car road road2d dynamic-obst single-gate double-gates double-gates+

Metrics: shield invocations per episode safety violations per episode total return in final 10 episodes

Baselines: MPS REVEL CPO PPO-Lag TD3

Builds on: Model Predictive Shielding TD3 Monte Carlo Tree Search REVEL

Closest papers by meaning

• Probabilistic Shielding for Safe Reinforcement Learning (AAAI 2025 · similarity 0.89)
• Explainably Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• Safety Representations for Safer Policy Learning (ICLR 2025 · similarity 0.88)
• A Provable Approach for End-to-End Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
• Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty (NeurIPS 2026 · similarity 0.86)
• Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learning (NeurIPS 2026 · similarity 0.86)