Research map / Reinforcement learning / Offline RL
Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Diffusion-DICE, which reinterprets DICE as a transformation from behavior to optimal policy distribution, learns the transformation's guidance term with in-sample learning inside a diffusion model, and selects among candidate actions with the DICE critic (guide-then-select).
Problem
Offline RL policy extraction: Gaussian policies limit expressivity of DICE-based methods, and existing diffusion-based offline RL methods query critic values on OOD diffusion-generated actions, causing error exploitation.
Approach
View DICE's optimal stationary distribution ratio as transforming the behavior policy into the optimal policy; decompose the optimal policy score into the behavior score plus a guidance gradient learned in-sample (In-sample Guidance Learning) with a piecewise f-divergence for stability, then select the highest-Q action from a few candidates.
Key results
- Diffusion-DICE reports 60.0 on halfcheetah-medium, 100.2 on hopper-medium, and 98.1 on antmaze-umaze, outperforming baselines on 13 of 15 D4RL tasks in the main table.
- On kitchen-partial and pen-cloned it reaches 78.3 and 83.8, far above Diffusion-QL (60.5, 57.3).
- A guidance-free variant is shown to have higher overestimated Q values and lower normalized return than the full guide-then-select method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Diffusion-DICE instead of QGPO | D4RL offline RL · halfcheetah-medium | normalized score: 60.0 ±0.6 vs 54.1 ±0.4 | Helped |
| Use Diffusion-DICE instead of QGPO | D4RL offline RL · hopper-medium | normalized score: 100.2 ±3.2 vs 98.0±2.6 | Helped |
| Use Diffusion-DICE instead of Diffusion-QL | D4RL offline RL · walker2d-medium-replay | normalized score: 90.8 ±2.6 vs 95.5 ±1.5 | Hurt |
| Use Diffusion-DICE instead of SfBC | D4RL offline RL · antmaze-umaze-diverse | normalized score: 82.0 ±8.4 vs 85.3 ±3.6 | Hurt |
| Use Diffusion-DICE instead of QGPO | D4RL offline RL · antmaze-medium-play | normalized score: 91.3 ±3.1 vs 83.6±4.4 | Helped |
| Use Diffusion-DICE instead of Diffusion-QL | D4RL offline RL · kitchen-partial | normalized score: 78.3 ±3.4 vs 60.5±6.9 | Helped |
| Use Diffusion-DICE instead of QGPO | D4RL offline RL · pen-human | normalized score: 84.4 ±2.7 vs 73.9 ±4.6 | Helped |
| Add the guide stage before selecting candidate actions (ablation) | Diffusion-DICE · D4RL evaluation | normalized score | Helped |
| Add the guide stage before selecting candidate actions (ablation) | Diffusion-DICE · D4RL evaluation | average Q(s, a) overestimation | Helped |
Limitations
- Diffusion sampling is costly and slow.
- Diffusion model training may suffer in low-data regimes.
Experimental setup
Datasets: D4RL
Benchmarks: D4RL
Metrics: normalized score
Baselines: CQL IQL SQL O-DICE Diffusion-QL SfBC QGPO IDQL
Builds on: DICE Diffusion-QL IDQL QGPO CEP
Closest papers by meaning
- ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update (ICLR 2024 · similarity 0.92)
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
- Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.90)
- Diffusion Policies Creating a Trust Region for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.89)
- Relaxed Stationary Distribution Correction Estimation for Improved Offline Policy Optimization (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.