Research map / Reinforcement learning / Policy gradient and actor-critic
SafeDreamer: Safe Reinforcement Learning with World Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SafeDreamer, a DreamerV3-based safe model-based RL method that combines online/background safety-reward planning with Lagrangian methods to reach near-zero-cost performance on Safety-Gymnasium vision and low-dimensional tasks.
Problem
Safe RL methods relying on cost functions often fail to achieve zero-cost performance, especially with vision-only inputs, due to world model inaccuracies and poor sample efficiency.
Approach
Integrates Lagrangian methods into world-model planning: online safety-reward planning with the Constrained Cross-Entropy Method (OSRP, OSRP-Lag with PID Lagrangian) and background planning that trains a safe actor with an augmented Lagrangian loss (BSRP-Lag), all built on DreamerV3.
Key results
- SafeDreamer (BSRP-Lag) achieves a 94.3% reduction in costs versus LAMBDA-level reward on Safety-Gymnasium vision tasks, reaching nearly zero-cost performance.
- On FormulaOne, SafeDreamer (BSRP-Lag) reaches cost return 0.43 vs DreamerV3's 56.36, with reward 25.3 vs 32.2.
- On Car-Racing, SafeDreamer (BSRP-Lag) reaches cost return 0.04 with reward 763.1, versus DreamerV3 cost 580.1 and reward 775.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SafeDreamer (BSRP-Lag) instead of PPO-Reward Shaping | SafeDreamer on MetaDrive · MetaDrive | Episode Cost Return: 14.1 ± 6.2 -> 2.2 ± 0.1 | Helped |
| Use SafeDreamer (BSRP-Lag) instead of DreamerV3 | SafeDreamer on Car-Racing · Car-Racing | Average Cost Return: 580.1 ± 2.35 -> 0.04 ± 0.01 | Helped |
| Use SafeDreamer (BSRP-Lag) instead of DreamerV3 | SafeDreamer on Car-Racing · Car-Racing | Average Reward Return: 775.1 ± 0.56 -> 763.1 ± 1.34 | Hurt |
| Use SafeDreamer (BSRP-Lag) instead of DreamerV3 | SafeDreamer on FormulaOne Level 1 · FormulaOne Level 1 | Average Cost Return: 56.36 ± 2.32 -> 0.43 ± 0.08 | Helped |
| Use SafeDreamer (BSRP-Lag) instead of DreamerV3 | SafeDreamer on FormulaOne Level 1 · FormulaOne Level 1 | Average Reward Return: 32.2 ± 1.22 -> 25.3 ± 0.51 | Hurt |
| Use OSRP instead of BSRP-Lag (ablation) | SafeDreamer · SafetyPointButton1 | reward | Helped |
| Use BSRP-Lag instead of OSRP (ablation) | SafeDreamer · SafetyPointPush1 | reward | Helped |
| Add the cost critic to estimate cost return (OSRP-Lag instead of OSRP) (ablation) | SafeDreamer · SafetyPointGoal2 | cost | Mixed |
Surprising findings
- BSRP-Lag converged to an optimal wedge-the-head push policy in SafetyPointPush1 that OSRP variants struggled to find within finite planning horizons.
- Model-free baselines in the paper were given a cost limit of 2.0 instead of the conventional 25.0, which the authors note highlights difficulty of model-free methods under tight budgets.
Limitations
- SafeDreamer trains each task independently, incurring substantial compute cost per task.
- Cost threshold was set to 2.0 rather than zero for vision tasks because of cost model and critic estimation errors.
- Online planning is time-consuming and can be overly conservative or struggle to find dexterous strategies within finite horizons.
Experimental setup
Datasets: Safety-Gymnasium SafetyPointGoal1 SafetyPointGoal2 SafetyPointPush1 SafetyPointButton1 SafetyCarGoal1 MetaDrive Car-Racing FormulaOne
Benchmarks: Safety-Gymnasium MetaDrive Car-Racing FormulaOne
Metrics: average undiscounted reward return average undiscounted cost return cost rate IQM normalized reward and cost
Baselines: PPO-Lag TRPO-Lag CPO Sauté PPO Sauté TRPO Safe SLAC LAMBDA MPC:sim MBPPO-Lag DreamerV3
Builds on: DreamerV3 PID Lagrangian Constrained Cross-Entropy Method LAMBDA Safe SLAC
Closest papers by meaning
- FOSP: Fine-tuning Offline Safe Policy through World Models (ICLR 2025 · similarity 0.89)
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer (NeurIPS 2025 · similarity 0.88)
- Long-Term Safe Reinforcement Learning with Binary Feedback (AAAI 2024 · similarity 0.88)
- SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning (NeurIPS 2025 · similarity 0.88)
- A Provable Approach for End-to-End Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.