Research map / Reinforcement learning / Policy gradient and actor-critic
Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QVPO, a model-free online RL algorithm that trains diffusion policies with a Q-weighted variational lower-bound loss, a diffusion entropy regularizer, and K-sample action selection for behavior policy, reporting state-of-the-art MuJoCo returns.
Problem
Applying diffusion policies in online model-free RL is hard because the diffusion variational lower bound needs good (optimal) action samples that are unavailable online, and diffusion policies lack tractable log-likelihoods for entropy regularization and have high variance.
Approach
Weights the diffusion VLO loss by Q-derived weights (qadv: positive advantage estimated from sampled actions) so that the weighted bound is a tight lower bound of the RL policy objective; adds an entropy regularizer that trains the diffusion model on uniform-sampled actions, and selects the best of K sampled actions for behavior policy and smaller K for TD targets.
Key results
- QVPO reaches 3728.5 on Hopper-v3, 5191.8 on Walker2d-v3, 6425.1 on Ant-v3, and 11385.6 on HalfCheetah-v3 after 1e6 steps, the best among compared methods on these four tasks.
- On Humanoid-v3, QVPO (5306.6) is comparable to TD3 (5353.5), which is bolded as best in the table.
- QVPO with 20 diffusion steps and entropy regularization approaches the performance of 100 diffusion steps without it, per the Ant-v3 ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QVPO instead of the strongest listed baseline (DIPO) | MuJoCo online RL · Hopper-v3 | episodic reward | Helped |
| Use QVPO instead of the strongest listed baseline (SAC) | MuJoCo online RL · Walker2d-v3 | episodic reward | Helped |
| Use QVPO instead of the strongest listed baseline (DIPO) | MuJoCo online RL · Ant-v3 | episodic reward | Helped |
| Use QVPO instead of the strongest listed baseline (SAC) | MuJoCo online RL · HalfCheetah-v3 | episodic reward | Helped |
| Use QVPO instead of the strongest listed baseline (TD3) | MuJoCo online RL · Humanoid-v3 | episodic reward | Hurt |
| Add diffusion entropy regularization (ablation) | QVPO with 20 diffusion steps · Ant-v3 | episodic reward | Helped |
| Use 20 instead of 100 diffusion steps without entropy regularization (ablation) | QVPO · Ant-v3 | episodic reward | Hurt |
| Increase behavior-policy action selection from K_b=1 to K_b=4, keeping K_t=1 (ablation) | QVPO · Ant-v3 | episodic reward | Helped |
| Set target-policy action selection K_t=2 rather than K_t=4 with K_b=4 (ablation) | QVPO · Ant-v3 | training stability | Helped |
| Increase behavior-policy action selection to K_b=20 (with K_t=2) (ablation) | QVPO · Ant-v3 | exploration | Hurt |
Surprising findings
- DIPO performed worse with 20 diffusion steps, so 100 steps were used for DIPO while QVPO used 20.
- Standard SAC/TD3 on Humanoid-v3 match or beat QVPO in the reported table, showing the advantage is not universal.
Limitations
- Entropy regularization is a heuristic approximation and cannot adaptively tune the entropy weight or be included in the TD target as in SAC.
- Inference is about 30x slower than SAC per action, and training time is higher than standard baselines.
- Q-weighted loss is a tight lower bound only under certain conditions, and the qcut/qadv transformations are approximations.
Experimental setup
Datasets: MuJoCo Ant-v3 MuJoCo HalfCheetah-v3 MuJoCo Hopper-v3 MuJoCo Humanoid-v3 MuJoCo Walker2d-v3
Benchmarks: MuJoCo Hopper-v3 MuJoCo Walker2d-v3 MuJoCo Ant-v3 MuJoCo HalfCheetah-v3 MuJoCo Humanoid-v3
Metrics: episodic reward sample efficiency training time inference time
Baselines: PPO SPO TD3 SAC DIPO QSM
Builds on: DDPM DIPO Diffusion-QL SAC
Closest papers by meaning
- Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching (NeurIPS 2026 · similarity 0.91)
- Diffusion Policies Creating a Trust Region for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient (NeurIPS 2024 · similarity 0.90)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.90)
- Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.