Research map / Reinforcement learning / Visual and representation RL
SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPiDR, a domain-randomization method that penalizes costs with an ensemble-variance estimate of sim-to-real uncertainty, giving a provable zero-shot safety bound for CMDP transfer.
Problem
Policies trained under domain randomization in simulators can violate safety constraints on the real system because the sim-to-real gap causes costs to be underestimated; robust safe RL methods that guarantee safety are hard to scale.
Approach
Bound the real-world cost by the simulated cost plus a Wasserstein-based discrepancy term, then approximate that term with the component-wise variance of next-state predictions across an ensemble of randomized dynamics, and add lambda times this variance to the cost while keeping the standard domain randomization pipeline unchanged.
Key results
- On real-world RaceCar and Unitree Go1, SPiDR satisfies the constraint while domain randomization dramatically violates it even when it satisfies the constraint in simulation.
- RAMU succeeds in following commands in 12 ± 4.8% of Go1 trials, while SPiDR completes all trials without falling.
- SPiDR satisfies constraints on all six sim-to-sim tasks, while Nominal and Domain Randomization fail; for n ≥ 8 ensemble members safety is maintained with marginal runtime overhead.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add uncertainty-based cost penalty | SPiDR versus Domain Randomization · RaceCar and Unitree Go1 real-world transfer | constraint satisfaction | Helped |
| Add uncertainty-based cost penalty | SPiDR versus Domain Randomization · RaceCar real-world transfer | objective performance | Helped |
| Use SPiDR instead of RAMU | Unitree Go1 · Unitree Go1 real-world trials | command-following success: 12 \pm 4.8\% | Helped |
| Add uncertainty-based cost penalty | SPiDR versus Nominal and Domain Randomization · Six sim-to-sim tasks | constraint satisfaction | Helped |
| Add uncertainty-based cost penalty | SPiDR versus RAMU · RWRL environments | objective performance | No clear effect |
| Add uncertainty-based cost penalty | SPiDR versus RAMU · RaceCar | objective performance | Helped |
| Increase ensemble size to n ≥ 8 (ablation) | SPiDR · WalkerWalk, QuadrupedRun, and HumanoidWalk | constraint satisfaction | Helped |
| Increase ensemble size (ablation) | SPiDR · WalkerWalk, QuadrupedRun, and HumanoidWalk | objective performance | No clear effect |
| Use SPiDR with image observations and privileged state for cost penalty | DrQ + CRPO · CartpoleSwingup vision evaluation | constraint satisfaction | Helped |
| Set λ to 0.6 (ablation) | SPiDR · CartpoleBalance across actuator gear settings | constraint satisfaction | Helped |
| Set λ to 0.6 (ablation) | SPiDR · CartpoleBalance across actuator gear settings | objective performance | Hurt |
| Enlarge the randomized throttle range (ablation) | Domain Randomization · RaceCar | constraint satisfaction | No clear effect |
Surprising findings
- Domain randomization is unsafe even when the simulated constraint is satisfied, and enlarging the randomization range of the RaceCar throttle did not restore safety.
- On RaceCar, domain randomization overshoots the target, so the more conservative SPiDR policy also achieves higher objective.
Limitations
- Assumes the problem is feasible in simulation; no guarantee if infeasible without simulator improvement or safe online learning.
- Penalty with the variance estimator can reduce the feasible set and cost some performance; the bound's tightness depends on assumptions like Lipschitz continuity and a mild sim-to-real gap.
- Real-world RaceCar experiments depend on licensed motion-capture software and are hard to reproduce.
Experimental setup
Datasets: RWRL benchmark Safety Gym RaceCar environment DeepMind Control Suite
Benchmarks: RWRL Safety Gym RaceCar DeepMind Control locomotion tasks (WalkerWalk, QuadrupedRun, HumanoidWalk) Unitree Go1 real-world RaceCar real-world
Metrics: objective return constraint cost vs budget training runtime relative to DR constraint satisfaction rate on real hardware
Baselines: Nominal Domain Randomization RAMU
Builds on: domain randomization constrained MDP CRPO RAMU SAC
Closest papers by meaning
- Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL (NeurIPS 2024 · similarity 0.89)
- Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guarantees (Abstract Reprint) (AAAI 2024 · similarity 0.89)
- SafeDreamer: Safe Reinforcement Learning with World Models (ICLR 2024 · similarity 0.88)
- Long-Term Safe Reinforcement Learning with Binary Feedback (AAAI 2024 · similarity 0.87)
- A Provable Approach for End-to-End Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.