Research map / Reinforcement learning / Offline RL
Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FISOR, which enforces hard state-wise safety constraints in safe offline RL by identifying the largest feasible region via an offline HJ-reachability feasible value function, then extracts a policy as weighted behavior cloning with a time-independent energy-guided diffusion model.
Problem
Safe offline RL usually enforces soft (expected) cost constraints, which permit unsafe outcomes; enforcing hard zero-violation constraints offline is hard because safety, reward, and behavior regularization are tightly coupled and unstable.
Approach
Use HJ reachability to convert the hard constraint into identifying the largest feasible region from offline data (via reversed expectile regression on a feasible Bellman operator), decouple reward maximization in feasible states and violation minimization in infeasible states, and extract the optimal policy as weighted behavior cloning with a diffusion model whose weighted regression loss realize
Key results
- FISOR is the only method meeting the safety requirement (normalized cost < 1) in all tasks, and achieves the highest safe-agent reward in most of the 26 DSRL tasks.
- On the MetaDrive average, FISOR reaches normalized reward 0.36 with cost 0.25, versus CDT at 0.25/1.45 and TREBI at 0.13/3.18.
- Ablation: removing HJ reachability (using cost value instead of feasible value) reduces safety-performance, e.g., CarRun reward drops from 0.73 to -0.33 with cost 0.00 and DroneRun cost rises to 5.12.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HJ reachability with a feasible value function instead of cost value function to identify feasibility (ablation) | FISOR · CarRun | normalized return: -0.33 -> 0.73 | Helped |
| use HJ reachability with a feasible value function instead of cost value function to identify feasibility (ablation) | FISOR · CarRun | normalized cost: 0.00 -> 0.14 | Hurt |
| use HJ reachability with a feasible value function instead of cost value function to identify feasibility (ablation) | FISOR · BulletGym tasks | normalized return | Helped |
| set the weight for the infeasible region to 0 (ablation) | FISOR · DSRL tasks | normalized cost | Hurt |
| replace the diffusion policy with a conventional Gaussian policy (ablation) | FISOR · DSRL tasks | normalized cost | Hurt |
| use FISOR instead of CDT | safe offline RL · SafetyGym Average | normalized cost: 4.31 -> 0.35 | Helped |
| use FISOR instead of BC | safe offline RL · BulletGym Average | normalized cost: 10.58 -> 0.10 | Helped |
| use FISOR instead of CDT | safe offline RL · MetaDrive Average | normalized cost: 1.45 -> 0.25 | Helped |
| use FISOR instead of CDT | safe offline RL · SafetyGym Average | normalized return: 0.50 -> 0.30 | Hurt |
| use FISOR instead of BC | safe offline RL · BulletGym Average | normalized return: 0.72 -> 0.39 | Hurt |
| use FISOR instead of COptiDICE | safe offline RL · MetaDrive Average | normalized return: 0.63 -> 0.36 | Hurt |
Surprising findings
- Removing HJ reachability produces overly conservative policies on several tasks (e.g., CarRun reward -0.33 vs 0.73 for FISOR), showing approximation error in the cost-based infeasible region hurts returns.
- Soft-constraint baselines are highly sensitive to the chosen cost limit, sometimes increasing final cost when the limit is made smaller.
Limitations
- Limited offline data size can hurt performance.
- Evaluated only on DSRL benchmark tasks with tuned cost limits; no real-world validation.
- Hard constraints can induce policy conservatism, and the approach relies on approximation of feasible value via offline Bellman operator.
Experimental setup
Datasets: DSRL benchmark datasets
Benchmarks: DSRL Safety-Gymnasium Bullet-Safety-Gym MetaDrive
Metrics: normalized return normalized cost
Baselines: BC CDT BCQ-Lag CPQ COptiDICE TREBI
Builds on: IQL HJ reachability Diffusion-QL CPQ DSRL
Closest papers by meaning
- Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning (AAAI 2025 · similarity 0.91)
- Offline Safe Reinforcement Learning Using Trajectory Classification (AAAI 2025 · similarity 0.91)
- Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL (NeurIPS 2025 · similarity 0.90)
- FOSP: Fine-tuning Offline Safe Policy through World Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.