Research map / Reinforcement learning / Offline RL

Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL

NeurIPS 2025 · Reinforcement learning › Offline RL · approach: Behavior regularization for offline RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DRCORL, an offline constrained RL method that pretrains a diffusion behavior policy, regularizes a Gaussian policy toward it via reverse KL using the diffusion score, and balances reward and cost with gradient manipulation and UCB cost critics.

Problem

Offline safe reinforcement learning must maximize return under soft cost constraints without querying the environment, while avoiding out-of-distribution actions and reward-safety conflicts.

Approach

Use a diffusion model to capture the behavior policy, distill it into a fast Gaussian policy by regularizing with the reverse KL computed from the diffusion score function (no diffusion sampling at inference), and switch/combine reward and cost gradients with slack-band gradient manipulation using pessimistic UCB cost critics.

Key results

• On CarGoal1, DRCORL reaches normalized reward 0.91 with normalized cost 0.00, while all baselines in that row except BC-Safe and COptiDICE-free rows exceed cost 1 or have lower reward among safe methods.
• DRCORL keeps normalized cost at or below 1 across the three cost limits (10, 20, 30) on the ablation, with zero violations reported.
• Authors report faster inference than diffusion/transformer-based methods (TREBI, FISOR) on 1,000 action samples in HalfCheetah, though MLP methods like BCQ and CPQ are fastest.

Measured effects reported in the paper

Surprising findings

• The paper's main Table reports a cost of 0.89 on AntVel and 0.79 on HalfCheetahVel-like rows, with several tasks (e.g., PointGoal2 cost 0.00 but PointButton1 cost 0.40) showing close-to-threshold costs, which is less clean than the headline claims.
• The theoretical convergence analysis drops the reverse KL term, so the guarantees do not cover the full DRCORL algorithm.

Limitations

• Pretraining (diffusion model plus critics) incurs substantial computational overhead.
• Zero constraint violation is hard to guarantee when dataset quality and coverage vary.
• Theoretical analysis is for the tabular setting and omits the reverse KL regularizer.

Experimental setup

Datasets: DSRL offline safe RL benchmark (Safety-Gymnasium and Bullet-Safety-Gym tasks)

Benchmarks: DSRL Safety-Gymnasium Bullet-Safety-Gym

Metrics: normalized return normalized cost inference time

Baselines: BC-Safe BEARL BCQ-Lag CPQ COptiDICE CDT CAPS CCAC TREBI FISOR

Builds on: DSRL Implicit Q-Learning CRPO Diffusion-QL CPO

Closest papers by meaning

• Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.90)
• Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model (ICLR 2024 · similarity 0.90)
• Offline Safe Reinforcement Learning Using Trajectory Classification (AAAI 2025 · similarity 0.90)
• Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning (AAAI 2025 · similarity 0.90)
• Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• FOSP: Fine-tuning Offline Safe Policy through World Models (ICLR 2025 · similarity 0.89)