Research map / Reinforcement learning / Offline RL
Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BDPO, a behavior-regularized actor-critic for diffusion policies that penalizes pathwise KL summed over single-step reverse diffusion transitions, trained with two-time-scale TD learning, and evaluated on 2D synthetic tasks and D4RL.
Problem
Behavior regularization in offline RL is well studied for Gaussian or deterministic policies, but it is unclear how to apply it to expressive diffusion policies because their action log-probabilities are intractable.
Approach
Compute the KL regularization as accumulated discrepancies of single-step reverse diffusion transitions (pathwise KL), which are analytically tractable Gaussians, and optimize the diffusion actor with per-step diffusion value functions in a two-time-scale actor-critic, avoiding back-propagation through the full diffusion chain.
Key results
- BDPO reaches 856.7 total on the 9 locomotion datasets vs 836.4 for DAC and 791.2 for Diffusion-QL (table reports sums).
- BDPO antmaze total 519.6 vs 459.9 for DAC, the best baseline in the table.
- Runtime with N=50 diffusion steps: BDPO 303 min total vs Diffusion-QL 560 min.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use diffusion actor parameterization instead of a deterministic policy (ablation) | BDPO · D4RL locomotion tasks | normalized score | Helped |
| Use diffusion actor parameterization instead of a Gaussian policy (ablation) | BDPO · D4RL locomotion tasks | normalized score | Helped |
| Use BDPO rather than the strongest listed baseline | D4RL locomotion tasks · halfcheetah-medium-v2; halfcheetah-medium-replay-v2; halfcheetah-medium-expert-v2; walker2d-medium-expert-v2 | normalized score | Helped |
| Use BDPO rather than the strongest listed baseline | D4RL locomotion tasks · hopper-medium-v2; hopper-medium-replay-v2; walker2d-medium-v2; walker2d-medium-replay-v2; hopper-medium-expert-v2 | normalized score | Hurt |
| Use BDPO rather than the strongest listed baseline | D4RL antmaze tasks · antmaze-medium-play-v0; antmaze-medium-diverse-v0; antmaze-large-play-v0; antmaze-large-diverse-v0 | normalized score | Helped |
| Use BDPO rather than the strongest listed baseline | D4RL antmaze tasks · antmaze-umaze-v0; antmaze-umaze-diverse-v0 | normalized score | Hurt |
Surprising findings
- Gaussian and deterministic actor parameterizations underperform the diffusion actor at convergence, even after tuning eta per variant.
Limitations
- Relies on the Gaussian single-step approximation of reverse transitions; exact Gaussianity holds only in the continuous limit.
- Fixed regularization strength eta and LCB coefficient rho tuned per dataset; auto-tuning was found to cause fluctuations.
- Antmaze results show higher variance on some datasets due to sparse rewards.
Experimental setup
Datasets: D4RL halfcheetah/hopper/walker2d -v2 D4RL antmaze -v0 synthetic 2D datasets (8gaussians, 2spirals, moons)
Benchmarks: D4RL locomotion-v2 D4RL antmaze-v0 synthetic 2D datasets
Metrics: normalized score runtime (minutes)
Baselines: CQL IQL Decision Diffuser SfBC IDQL QGPO SRPO DTQL Diffusion-QL EDP DAC
Builds on: Diffusion-QL DAC DPPO soft actor-critic Girsanov theorem
Closest papers by meaning
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.94)
- Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.93)
- Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.92)
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
- An Implicit Trust Region Approach to Behavior Regularized Offline Reinforcement Learning (AAAI 2024 · similarity 0.91)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.