Research map / Reinforcement learning / Offline RL
What Makes a Good Diffusion Planner for Decision Making?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic empirical study (over 6,000 diffusion planners) of design choices for diffusion planning in offline RL, distilling them into a simple strong baseline, Diffusion Veteran (DV).
Problem
Design choices for diffusion planners in offline RL (guidance, backbone, action generation, planning stride) are inconsistent across prior work and their effects are unclear.
Approach
Controlled sweep over key components of diffusion planners (guided sampling, denoising backbone, action generation, planning stride, network depth) and distillation into a baseline DV: Transformer denoiser, separate state-planning with inverse dynamics, jump-step planning, and Monte Carlo sampling with selection instead of guidance.
Key results
- DV achieves an average normalized score of 163.6 on Maze2D, 83.2 on AntMaze and 83.8 on Kitchen, the best among compared diffusion planners and policies in the paper's table.
- Transformer denoiser with 2.64M parameters reaches 69.5 average on Adroit, while U-Net needs up to 63.11M parameters to reach 67.8.
- Monte Carlo sampling with selection (MCSS) reaches 69.5 average on Adroit, above CFG (67.7), CG (62.0) and unguided sampling (60.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use separate state planning with inverse dynamics rather than joint state-action generation (ablation) | Diffusion planning · Adroit Hand | average score: 67.9 -> 69.5 | Helped |
| increase planning stride from 1 to 2 (ablation) | Diffusion planning · Adroit Hand | average score: 68.8 -> 69.5 | Helped |
| replace U-Net with Transformer as denoising backbone (ablation) | Diffusion planner · Adroit Hand | average score: 67.8 -> 69.5 | Helped |
| increase Transformer depth from 1 to 2 layers (ablation) | Diffusion planner · Adroit Hand | average score: 23.3 -> 69.5 | Helped |
| increase Transformer depth from 2 to 3 layers (ablation) | Diffusion planner · Adroit Hand | average score: 69.5 -> 68.1 | Hurt |
| replace MCSS with classifier-free guidance (ablation) | Diffusion planner · Adroit Hand | average score: 69.5 -> 67.7 | Hurt |
| replace MCSS with classifier guidance (ablation) | Diffusion planner · Adroit Hand | average score: 69.5 -> 62.0 | Hurt |
| replace MCSS with unguided sampling without selection (ablation) | Diffusion planner · Adroit Hand | average score: 69.5 -> 60.9 | Hurt |
| use diffusion planning rather than diffusion policy | Decision-making methods · Kitchen, AntMaze, and Maze2D | normalized return | Helped |
| use diffusion policy rather than diffusion planning | Decision-making methods · MuJoCo locomotion | return | Helped |
Surprising findings
- Unconditional sampling with learned-critic selection outperforms classifier and classifier-free guidance when the dataset has many near-optimal trajectories.
- Larger models do not necessarily improve diffusion planners; a 3-layer Transformer slightly underperforms a 2-layer one on Adroit.
Limitations
- Study restricted to state-based robotics tasks in D4RL; vision-based, goal-conditioned, and partially observable settings are not explored.
- Diffusion planners have substantial computational cost.
- Evaluation uses a single performance metric (total return), ignoring interpretability and safety.
Experimental setup
Datasets: D4RL Kitchen D4RL AntMaze D4RL Maze2D D4RL MuJoCo Adroit Hand
Benchmarks: D4RL Kitchen D4RL AntMaze D4RL Maze2D Adroit Hand
Metrics: normalized average episode return
Baselines: Diffuser Decision Diffuser HD DQL IDQL AdaptDiffuser CQL IQL BCQ SfBC CEP
Builds on: Diffuser Decision Diffuser CleanDiffuser classifier-free guidance IDQL
Closest papers by meaning
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Habitizing Diffusion Planning for Efficient and Effective Decision Making (ICML 2025 · similarity 0.89)
- Simple Hierarchical Planning with Diffusion (ICLR 2024 · similarity 0.89)
- Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.88)
- Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- DiffuserLite: Towards Real-time Diffusion Planning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.