Research map / Reinforcement learning / Offline RL
Diffusion Policies Creating a Trust Region for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DTQL, a dual-policy offline RL method where a diffusion policy does pure behavior cloning and a one-step Gaussian policy is trained with a diffusion trust region loss plus Q-maximization, removing iterative denoising from training and inference.
Problem
Diffusion-policy offline RL (e.g., Diffusion Q-Learning) is computationally expensive because action generation requires iterative denoising during both training and inference, and existing acceleration methods often degrade performance.
Approach
A diffusion trust region loss, derived from the ELBO of a fixed diffusion behavior-cloning model, trains a one-step policy to produce actions with high conditional likelihood under the behavior data, creating a mode-seeking trust region instead of KL-based distribution matching (SDS/VSD/DMD-style distillation).
Key results
- On D4RL Gym average, DTQL scores 88.7 versus 88.0 for DQL and 87.1 for SRPO; on AntMaze-v0 average it reaches 73.6 versus 69.6 for DQL and 79.1 for IDQL (IDQL higher).
- Inference time per antmaze-umaze-v0 trajectory is 0.35 s for DTQL versus 3.03 s for DQL and 3.04 s for IDQL, while total training time is 3.33 h versus 6.70 h (DQL) and 14.64 h (IDQL).
- On Gym medium tasks, the diffusion trust region loss outperforms the KL behavior-distillation loss (e.g., hopper-medium 99.6 vs 15.0, walker2d-medium 89.4 vs 3.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DTQL instead of DQL | D4RL Adroit Average · Adroit Average | normalized score: 65.1 -> 72.7 | Helped |
| Use DTQL instead of IDQL | D4RL Antmaze v0 · Antmaze Average (v0) | normalized score: 79.1 -> 73.6 | Hurt |
| Use DTQL instead of DQL | D4RL Kitchen · Kitchen Average | normalized score: 69.0 -> 71.8 | Helped |
| Use DTQL instead of DQL | D4RL Gym · Gym Average | normalized score: 88.0 -> 88.7 | Helped |
| Remove pretraining (ablation) | DTQL · halfcheetah-medium-v2 | normalized score: 57.9 -> 57.5 | No clear effect |
| Remove pretraining (ablation) | DTQL · hopper-medium-v2 | normalized score: 99.6 -> 87.6 | Hurt |
| Remove pretraining (ablation) | DTQL · walker2d-medium-v2 | normalized score: 89.4 -> 88.7 | No clear effect |
| Add the NLL term (ablation) | DTQL Gaussian policy · antmaze-umaze-v0 | final normalized score | No clear effect |
| Add the NLL term (ablation) | DTQL Gaussian policy · antmaze-large-diverse-v0 | final normalized score | Helped |
| Replace diffusion trust-region loss with KL behavior regularization (ablation) | DTQL · halfcheetah-medium-v2 | normalized score: 57.9 -> 24.1 | Hurt |
| Replace diffusion trust-region loss with KL behavior regularization (ablation) | DTQL · hopper-medium-v2 | normalized score: 99.6 -> 15.0 | Hurt |
| Replace diffusion trust-region loss with KL behavior regularization (ablation) | DTQL · walker2d-medium-v2 | normalized score: 89.4 -> 3.4 | Hurt |
Surprising findings
- On AntMaze-v0, IDQL (79.1) outperforms DTQL (73.6) on average, and DTQL's average falls below DQL-era numbers on some tasks, so gains are not uniform.
- The KL-based distillation variant collapses badly on Gym medium tasks (e.g., walker2d-medium 3.4), suggesting standard KL distillation is poorly suited for offline RL behavior regularization.
Limitations
- Performance is not uniformly best; IDQL outperforms DTQL on several AntMaze-v0 tasks and DQL on some Kitchen/Gym tasks.
- Experiments are limited to the offline setting; extension to online RL or image inputs is left open.
- The method relies on the EDM schedule, which the authors acknowledge does not satisfy the ELBO condition on which the theory is based.
Experimental setup
Datasets: D4RL halfcheetah D4RL hopper D4RL walker2d D4RL antmaze D4RL adroit D4RL kitchen
Benchmarks: D4RL Gym D4RL AntMaze D4RL Adroit D4RL Kitchen 2D bandit toy scenarios
Metrics: normalized score training time per 1k steps inference time per trajectory total wall time
Baselines: BC Onestep RL TD3+BC Decision Transformer CQL IQL DQL IDQL SRPO BCQ BEAR BRAC
Builds on: Diffusion Q-Learning Implicit Q-Learning EDM Score distillation sampling Diffusion-GAN
Closest papers by meaning
- Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.92)
- Learning a Diffusion Model Policy from Rewards via Q-Score Matching (ICML 2024 · similarity 0.91)
- Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.91)
- Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.