Research map / Reinforcement learning / Visual and representation RL
Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guarantees (Abstract Reprint)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sim-to-Lab-to-Real, combining HJ-reachability-based dual policies with value-based shielding and PAC-Bayes fine-tuning to give safety-aware, generalization-guaranteed visual navigation policies, validated in simulation and on a quadruped.
Problem
Safe RL policies for vision-based navigation often fail to generalize to novel environments and violate safety during training and deployment, and they lack certificates of performance and safety in unseen real environments.
Approach
Train a performance policy and a reachability-based backup (safety) policy with a Safety Bellman Equation, apply value-based shielding during Sim and Lab training, and fine-tune a latent-conditioned policy distribution in Lab so PAC-Bayes bounds certify success and safety in unseen environments.
Key results
- In Advanced-Realistic, the PAC-Bayes success bound with shielding improves from 0.366 to 0.786 (task) and safety bound from 0.367 to 0.794 relative to PAC-Bayes Control without shielding.
- Reachability-based safety critic reduces safety violations by up to 77% in Lab training and 38% in testing compared to baselines (in one reported setting).
- Real quadruped robot success reaches 0.767 in Advanced-Realistic for PAC+Shield+Perf versus 0.433 for PAC+Base and 0.667 for SQRL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add reachability-based shielding to the policy architecture | PAC Base · Advanced-Realistic | success bound: 0.297 to 0.701 | Helped |
| Add reachability-based shielding to the policy architecture | PAC Base · Advanced-Realistic | safety bound: 0.304 to 0.708 | Helped |
| Use reachability safety critic instead of risk-based safety critic | Safe RL methods · Lab training, five Lab setups | safety violations: 77%, 4%, 76%, 62%, and 23% | Helped |
| Use reachability safety critic instead of risk-based safety critic | Safe RL methods · Deployment, five Lab setups | safety violations: 38%, 26%, 54%, 34%, and 28% | Helped |
| Set shielding threshold to v_thr=0 instead of v_thr=-0.05 or v_thr=-0.10 (ablation) | PAC Shield Perf · Vanilla-Env Lab training | safety violations | Hurt |
| Set shielding threshold to v_thr=0 instead of v_thr=-0.05 or v_thr=-0.10 (ablation) | PAC Shield Perf · Vanilla-Env test | performance | Hurt |
| Use v_thr=-0.1 instead of v_thr=-0.05 (ablation) | PAC Shield Perf · Advanced-Realistic | average test success rate: 0.678 to 0.786 | Helped |
| Increase KL regularization weight from α=2 to α=5 (ablation) | PAC Shield Perf · Advanced-Dense, 500 Lab environments | success bound: 0.578 to 0.557 | Hurt |
| Increase KL regularization weight from α=2 to α=1 (ablation) | PAC Shield Perf · Advanced-Dense, 500 Lab environments | success bound: 0.578 to 0.512 | Hurt |
| Increase KL regularization weight to α=10 (ablation) | PAC Shield Perf · Lab training and test | testing performance | Hurt |
| Condition the backup policy on the latent variable (ablation) | PAC Shield Perf · Test environments | performance and safety | Hurt |
| Maximize trajectory diversity during Sim training (ablation) | PAC Shield Perf · Test environments | generalization performance | Helped |
Surprising findings
- Distribution-over-policies without shielding (PAC+Base, PAC+RP) performed worse than single-policy Base and RP in Advanced-Env, as high diversity causes frequent safety violations that hinder exploration.
- Adding a latent distribution to the backup policy (PAC+Shield+Both) gave similar or worse performance and safety than conditioning only the performance policy.
Limitations
- PAC-Bayes guarantees require at least about 100 or more Lab environments (N>=100, typically hundreds to thousands), which is resource-intensive.
- Backup policy can fail to ensure safety in tight, realistic indoor spaces, and performance is sensitive to the shielding threshold.
- Lab training uses simulated environments; real-environment Lab training is not evaluated.
Experimental setup
Datasets: 3D-FRONT
Benchmarks: Vanilla-Env (Vanilla-Normal, Vanilla-Dynamics, Vanilla-Task) Advanced-Env (Advanced-Dense, Advanced-Realistic) real indoor quadruped deployment
Metrics: safety violation ratio during Lab training success rate unfinished/failed trial fraction PAC-Bayes success and safety bounds
Baselines: Base Reward Penalty PAC-Bayes Control (PAC+Base) PAC+Reward Penalty SQRL Recovery RL Shield (ablation without latent distribution)
Builds on: Soft Actor-Critic PAC-Bayes Control reachability RL Recovery RL SQRL
Closest papers by meaning
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer (NeurIPS 2025 · similarity 0.89)
- SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning (NeurIPS 2025 · similarity 0.87)
- Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL (NeurIPS 2024 · similarity 0.87)
- A Physics-Informed Machine Learning Framework for Safe and Optimal Control of Autonomous Systems (ICML 2025 · similarity 0.86)
- Probabilistic Shielding for Safe Reinforcement Learning (AAAI 2025 · similarity 0.86)
- FOSP: Fine-tuning Offline Safe Policy through World Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.