Research map / Reinforcement learning / Offline RL
Prior-Guided Diffusion Planning for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prior Guidance, which learns a Gaussian prior over the DDIM latent noise of a behavior-cloned diffusion planner, optimized with a closed-form behavior-regularized objective using a latent-space value function, achieving single-sample guided planning in D4RL.
Problem
Guided sampling for diffusion planners in offline RL is suboptimal: classifier guidance yields suboptimal multi-modal actions, CFG generalizes poorly, and Monte Carlo sample selection is costly at inference and selects OOD trajectories when N is large.
Approach
Replace the standard Gaussian prior of a fixed behavior-cloned diffusion planner with a learnable Gaussian prior conditioned on state, trained with a behavior-regularized (f-divergence) objective evaluated in closed form between Gaussians, using a latent-space value function so no backpropagation through the denoising process is needed.
Key results
- PG reaches average normalized score 81.3 on Kitchen (vs. 74.8 DV*) and 83.4 on AntMaze (vs. 79.9 DV*), and 164.6 on Maze2D (vs. 158.5 DV*).
- PG with a single sample matches or exceeds MCSS with N=500 on average (84.6 MuJoCo, 83.4 AntMaze) while requiring far fewer samples at inference.
- PG without backpropagation through the denoising process achieves nearly identical scores to with backpropagation, while training substantially faster than DQL (planner), up to ~10x in AntMaze.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace DV* sample selection with Prior Guidance | Diffusion Veteran planner · Kitchen and Antmaze | normalized score | Helped |
| replace DV* sample selection with Prior Guidance | Diffusion Veteran planner · Maze2D | normalized score | Mixed |
| replace the uni-modal Gaussian prior with a multi-modal Gaussian mixture prior (ablation) | PG · MuJoCo average | normalized score: 84.6±0.2 vs 82.4±0.4 | Hurt |
| replace the uni-modal Gaussian prior with a multi-modal Gaussian mixture prior (ablation) | PG · Antmaze average | normalized score: 83.4±0.9 vs 85.8±0.6 | Helped |
| train the latent value function without backpropagation through the denoising process (ablation) | PG · D4RL benchmark | normalized score | No clear effect |
Surprising findings
- A uni-modal Gaussian prior outperforms a multi-modal Gaussian mixture prior on MuJoCo (84.6 vs 82.4), while the mixture helps slightly on AntMaze (85.8 vs 83.4).
Limitations
- Adds complexity through training both a latent value function and a learnable prior.
- Prior network architecture choices not thoroughly investigated.
- Evaluated only in offline settings; no high-dimensional image observations; DDIM bijectivity assumption fails with finite discretization steps.
Experimental setup
Datasets: D4RL MuJoCo D4RL Kitchen D4RL AntMaze D4RL Maze2D
Benchmarks: D4RL MuJoCo D4RL Kitchen D4RL AntMaze D4RL Maze2D
Metrics: normalized score average normalized score training time inference time
Baselines: BC CQL IQL SfBC DQL IDQL QGPO SRPO Diffuser Decision Diffuser AdaptDiffuser HDMI
Builds on: Diffusion Veteran Diffuser Diffusion Q-Learning SfBC DDIM
Closest papers by meaning
- Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.93)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.92)
- Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.92)
- Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.91)
- Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
- Learning to Reach Goals via Diffusion (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.