Research map / Reinforcement learning / Offline RL
FOSP: Fine-tuning Offline Safe Policy through World Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FOSP, a model-based offline-to-online safe RL method for vision-based tasks that combines in-sample critic learning, reachability-guided safe actor updates, and safe policy expansion for online fine-tuning.
Problem
Offline safe RL policies learned from static datasets generalize poorly and can violate safety constraints when deployed in unseen safety-constrained scenarios with vision inputs; naive online fine-tuning causes performance drops and violations.
Approach
Builds on DreamerV3/SafeDreamer world models with in-sample (IQL-style expectile) value learning for offline training, uses a reachability estimation function to unify hard feasibility and soft cost constraints in the actor update, and bridges to online fine-tuning via safe policy expansion that selects between the frozen offline policy and a new policy using normalized Q and cost-Q values.
Key results
- Across five vision-based Safety-Gymnasium tasks, FOSP's offline-to-online average cost falls to 0.54 with average reward 16.6, versus 0.91 for Recover-RL and 1.01 for SafeDreamer (average cost), while DreamerV3 reaches higher reward (20.8) but cost 171.1.
- Compared with online-only methods trained for 10M steps, FOSP (average reward 16.6, cost 0.54) outperforms CPO (14.5, 51.8) and PPO-Lagrangian (7.5, 27.7) on the same tasks.
- On real Franka robot transfer tasks with unseen obstacles and targets, FOSP fine-tuned over 40 trials reportedly completes tasks with zero constraint violations, while SafeDreamer sometimes violates constraints even when successful.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FOSP instead of Recover-RL | Offline-to-online safe RL · Safety-Gymnasium, five-task average | Reward return: 6.7 -> 16.6 | Helped |
| Use FOSP instead of DreamerV3 | Offline-to-online safe RL · Safety-Gymnasium, five-task average | Cost return: 171.1 -> 0.54 | Helped |
| Use FOSP instead of SafeDreamer | Offline-to-online safe RL · Safety-Gymnasium, five-task average | Reward return: 12.5 -> 16.6 | Helped |
| Use FOSP instead of SafeDreamer (+planning) | Online safe RL · Safety-Gymnasium, five-task average | Cost return: 1.77 -> 0.54 | Helped |
| Use FOSP instead of CPO | Online safe RL · Safety-Gymnasium, five-task average | Cost return: 51.8 -> 0.54 | Helped |
| Remove in-sample optimization (ablation) | FOSP · SafetyPointGoal2 | offline performance | Hurt |
| Directly fine-tune without safe policy expansion (ablation) | FOSP · SafetyPointGoal2 | transition stability | Hurt |
| Remove the reachability estimation function (ablation) | FOSP safe actor · SafetyPointGoal2 | overall performance | Hurt |
| Replace the Augmented Lagrangian penalty with the feasible value function (ablation) | FOSP · SafetyPointGoal2 | safety constraint compliance | Hurt |
| Increase the proportion of safe data (ablation) | FOSP offline dataset · Simulation tasks | reward return | Helped |
| Increase the proportion of safe data (ablation) | FOSP offline dataset · Simulation tasks | cost return | Hurt |
| Increase the proportion of unsafe data (ablation) | FOSP offline dataset · Simulation tasks | cost return | Helped |
Surprising findings
- Offline-to-online FOSP reportedly achieves lower final cost than SafeDreamer trained purely online for 0.5M steps, which the authors attribute to offline-learned safety constraints making the online model more sensitive to safety.
- Imbalanced dataset experiments show safe-heavy data yields aggressive policies with violations, while unsafe-heavy data yields overly conservative policies.
Limitations
- Real-world evaluation is limited to a single Franka SafeReach task family with three transfer variants and a small dataset of 200 trajectories.
- Relies on sparse manually-labeled costs and rewards for real-robot data.
- Simulation results are averaged over five seeds on a small set of Safety-Gymnasium tasks, and the method's gains depend on the offline dataset composition.
Experimental setup
Datasets: Safety-Gymnasium offline datasets (random, safe, unsafe policy rollouts) SafeReach real-robot teleoperation dataset
Benchmarks: Safety-Gymnasium PointGoal1 Safety-Gymnasium PointButton1 Safety-Gymnasium PointPush1 Safety-Gymnasium PointGoal2 Safety-Gymnasium CarGoal2 SafeReach real-robot (Franka)
Metrics: reward return cost return cost regret
Baselines: Recovery RL SafeDreamer DreamerV3 CPO PPO-Lagrangian
Builds on: DreamerV3 SafeDreamer PEX Implicit Q-learning RESPO
Closest papers by meaning
- Offline Safe Reinforcement Learning Using Trajectory Classification (AAAI 2025 · similarity 0.90)
- Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model (ICLR 2024 · similarity 0.89)
- Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL (NeurIPS 2025 · similarity 0.89)
- Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.89)
- SafeDreamer: Safe Reinforcement Learning with World Models (ICLR 2024 · similarity 0.89)
- Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.