Research map / Reinforcement learning / Policy gradient and actor-critic
Verified Safe Reinforcement Learning for Neural Network Dynamic Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VSRL, a safe RL method that trains neural network controllers for finite-horizon verified safety using curriculum learning with memorization, incremental verification via alpha-beta-CROWN, and multiple initial-state-dependent controllers.
Problem
Training RL controllers for nonlinear neural dynamical systems that are formally verified safe over a finite horizon K while maintaining high reward; existing safe RL methods give only empirical or asymptotic/forward-invariance guarantees that do not scale to nonlinear neural dynamics.
Approach
Combine a K-step curriculum (increasing verified horizon, memorizing near-unsafe regions in a buffer) with a differentiable reachability bound loss, incremental verification reusing bounds across multi-step intervals, and a set of initial-state-dependent controllers fine-tuned per unverified cluster.
Key results
- Verified 80-step safety in lane following with a single controller, versus at most K=8 for baselines.
- Verified 50-step safety in vehicle avoidance and 2D quadrotor (fixed and moving obstacles), versus best baseline K=13, 14, and 8 respectively.
- Verified 15-step safety in 3D quadrotor with 100% empirical safety over the full episode, versus best baseline K=4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VSRL instead of RESPO | Lane Following · Lane Following | Verified-80 (%): 99.8 -> 100.0 | Helped |
| Use VSRL instead of RESPO | Lane Following · Lane Following | Avg Reward: 383 \pm 7 -> 214 \pm 5 | Hurt |
| Use VSRL instead of MBPPO | Vehicle Avoidance (Moving Obstacles) · Vehicle Avoidance (Moving Obstacles) | Verified-50 (%): 82.6 -> 100.0 | Helped |
| Use VSRL instead of RESPO | Vehicle Avoidance (Moving Obstacles) · Vehicle Avoidance (Moving Obstacles) | Avg Reward: 391 \pm 20 -> 401 \pm 4 | Helped |
| Use VSRL instead of RESPO | 2D Quadrotor (Fixed Obstacles) · 2D Quadrotor (Fixed Obstacles) | Verified-50 (%): 60.4 -> 100.0 | Helped |
| Use VSRL instead of PPO-PID | 2D Quadrotor (Fixed Obstacles) · 2D Quadrotor (Fixed Obstacles) | Avg Reward: 411 \pm 25 -> 401 \pm 20 | Hurt |
| Use VSRL instead of CAP | 2D Quadrotor (Moving Obstacles) · 2D Quadrotor (Moving Obstacles) | Verified-50 (%): 57.1 -> 100.0 | Helped |
| Use VSRL instead of MBPPO | 2D Quadrotor (Moving Obstacles) · 2D Quadrotor (Moving Obstacles) | Avg Reward: 374\pm 6 -> 364\pm 4 | Hurt |
| Use VSRL instead of MBPPO | 3D Quadrotor (Fixed Obstacles) · 3D Quadrotor (Fixed Obstacles) | Verified-15 (%): 41.1 -> 100.0 | Helped |
| Use VSRL instead of PPO-PID | 3D Quadrotor (Fixed Obstacles) · 3D Quadrotor (Fixed Obstacles) | Avg Reward: 145 \pm 12 -> 122 \pm 14 | Hurt |
| Add incremental verification (ablation) | VSRL · Lane Following, 20-step training | Runtime (seconds; 20 epochs): 185.5 -> 24.8 | Helped |
| Add incremental verification (ablation) | VSRL · Vehicle Avoidance, 20-step training | Runtime (seconds; 20 epochs): 187.1 -> 25.5 | Helped |
Surprising findings
- Lane following shows a notable reward drop for VSRL (214 vs. about 326-383 for baselines), the only environment where reward decreases significantly.
Limitations
- Finite-horizon reachability gives weaker guarantees than forward invariance; safety is only verified for K steps.
- Assumes the transition function is a known ReLU neural network for verification.
- Empirical safety evaluated by sampling and verified horizons are short relative to episode length.
Experimental setup
Datasets: simulated control environments (lane following, vehicle avoidance, 2D quadrotor, 3D quadrotor)
Benchmarks: lane following vehicle avoidance 2D quadrotor fixed obstacles 2D quadrotor moving obstacles 3D quadrotor fixed obstacles
Metrics: Verified-K Verified-Max Emp-K Avg Reward
Baselines: PPO-Lag PPO-PID CAP MBPPO CBF-RL RESPO
Builds on: alpha-beta-CROWN auto_LiRPA PPO-Lagrangian CAP RESPO
Closest papers by meaning
- A Physics-Informed Machine Learning Framework for Safe and Optimal Control of Autonomous Systems (ICML 2025 · similarity 0.89)
- Safely Learning Controlled Stochastic Dynamics (NeurIPS 2025 · similarity 0.88)
- Provably Safe Neural Network Controllers via Differential Dynamic Logic (NeurIPS 2024 · similarity 0.88)
- Sampling-Based Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- Learning Verified Safe Neural Network Controllers for Multi-Agent Path Finding (AAAI 2025 · similarity 0.87)
- Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.