Research map / Reinforcement learning / Offline RL
OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OASIS, a constraint-conditioned diffusion model that synthesizes reshaped offline datasets so that a regularized offline safe RL agent (BCQ-Lag) can learn safer and more rewarding policies from imperfect data.
Problem
Offline safe RL with imperfect demonstrations (tempting or conservative datasets) suffers from the safe dataset mismatch problem, where regularization toward the behavior policy causes either safety violations or reward degradation.
Approach
Use a classifier-free-guided conditional diffusion model over sub-trajectories, conditioned on target reward and cost returns, to generate a new dataset shaped toward the cost threshold and high reward, then train a regularized offline safe RL agent (BCQ-Lag) on it, with theoretical TV-distance bounds on distribution shaping and constraint violation.
Key results
- On the tempting dataset, OASIS is the only method achieving a safe policy with high reward on most tasks, e.g., BallCircle reward 0.70 with normalized cost 0.45, CarCircle reward 0.76 with cost 0.89.
- OASIS achieves safe rewarding policies in BallRun (reward 0.28, cost 0.79) and DroneCircle (reward 0.60, cost 0.25), where most baselines violate constraints.
- Performance is robust to denoising steps K in {10, 20, 40}, and OASIS remains good with generated data ratio below 2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OASIS distribution shaping instead of FISOR | offline safe RL on BallRun (tempting) · BallRun (tempting) | normalized reward: 0.17 ± 0.03 -> 0.28 ± 0.01 | Helped |
| Use OASIS distribution shaping instead of CDT | offline safe RL on CarRun (tempting) · CarRun (tempting) | normalized cost: 0.67 ± 0.03 -> 0.02 ± 0.03 | Helped |
| Use OASIS distribution shaping instead of the best-reward baseline | offline safe RL on DroneRun (tempting) · DroneRun (tempting) | normalized reward: 0.84 ± 0.12 -> 0.13 ± 0.08 | Hurt |
| Use OASIS distribution shaping instead of CPQ | offline safe RL on BallCircle (tempting) · BallCircle (tempting) | normalized reward: 0.39 ± 0.34 -> 0.70 ± 0.01 | Helped |
| Use OASIS distribution shaping instead of CPQ | offline safe RL on CarCircle (tempting) · CarCircle (tempting) | normalized reward: 0.64 ± 0.02 -> 0.76 ± 0.03 | Helped |
| Use OASIS distribution shaping instead of FISOR | offline safe RL on DroneCircle (tempting) · DroneCircle (tempting) | normalized reward: 0.49 ± 0.05 -> 0.60 ± 0.01 | Helped |
| Vary denoising steps from K=10 to K=20 or K=40 (ablation) | OASIS on BallCircle · BallCircle | normalized reward: 0.71 ± 0.02; 0.70 ± 0.01; 0.71± 0.01 | No clear effect |
| Vary denoising steps from K=10 to K=20 or K=40 (ablation) | OASIS on BallRun · BallRun | normalized reward: 0.29 ± 0.04; 0.28 ± 0.01; 0.29 ± 0.01 | No clear effect |
Surprising findings
- CVAE-based data generation fails to incorporate conditions, producing near-identical datasets across different target conditions.
- Q-learning-based regularized methods such as BCQ-Lag and BEAR-Lag violate safety constraints on tempting datasets because regularization pulls the policy toward unsafe behavior.
Limitations
- Offline training takes longer due to dataset preprocessing and generation.
- Zero constraint violation remains hard to achieve with imperfect demonstrations.
- Evaluated only on Bullet-Safety-Gym locomotion tasks with a single constraint.
Experimental setup
Datasets: OSRL Bullet-Safety-Gym tasks with full, tempting, conservative, hybrid dataset variants
Benchmarks: Bullet-Safety-Gym BallRun CarRun DroneRun BallCircle CarCircle DroneCircle
Metrics: normalized reward normalized cost data efficiency robustness to denoising steps
Baselines: BC BCQ-Lag BEAR-Lag CPQ COptiDICE CDT FISOR CVAE-BCQL
Builds on: BCQ-Lag Diffuser Decision Diffuser classifier-free guidance OSRL
Closest papers by meaning
- Offline Safe Reinforcement Learning Using Trajectory Classification (AAAI 2025 · similarity 0.88)
- Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL (NeurIPS 2025 · similarity 0.88)
- Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model (ICLR 2024 · similarity 0.87)
- Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.87)
- FOSP: Fine-tuning Offline Safe Policy through World Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.