Research map / Reinforcement learning / Policy gradient and actor-critic
Efficient Online Reinforcement Learning for Diffusion Policy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Reweighted Score Matching (RSM), a reweighted denoising score matching loss that lets diffusion policies be trained in online RL from Q-values without sampling the optimal policy, yielding DPMD and SDAC algorithms that beat diffusion and classic RL baselines on MuJoCo.
Problem
Training expressive diffusion policies in online RL is hard because denoising score matching needs samples from the target (optimal) policy, which are unavailable, and backpropagating policy gradients through the diffusion chain is costly and unstable.
Approach
Generalize denoising score matching by reweighting the loss with a function of the noise-perturbed density so that the optimal score is unchanged, yielding tractable losses that need only Q-values and samples from the current policy or a chosen sampling distribution (for policy mirror descent and max-entropy policies).
Key results
- SDAC reaches 12210 on HalfCheetah and DPMD 6959 on Humanoid versus SAC 2858, with reported gains of 36.0% (HalfCheetah), 127.3% (Ant) and 143.5% (Humanoid) over SAC for the top variant.
- DPMD achieves 5683 on Ant versus 2500 for SAC and attains the best or near-best results on most of the 10 MuJoCo tasks.
- The RSM methods use roughly 1.1-1.2 GB GPU memory and about 16-21 minutes of training, lower than QVPO (5.2 GB) and DPPO (95 min).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DPMD instead of SAC | online RL policy on Humanoid · Humanoid | performance improvement: 143.5% | Helped |
| Use DPMD instead of SAC | online RL policy on Ant · Ant | performance improvement: 127.3% | Helped |
| Use the top-performing proposed variant instead of SAC | online RL policy on HalfCheetah · HalfCheetah | performance improvement: 36.0% | Helped |
| Use the top-performing proposed variant instead of SAC | online RL policy on Walker2d · Walker2d | performance improvement: 41.7% | Helped |
| Use DPMD instead of other diffusion-policy online RL baselines | online RL policy on Humanoid · Humanoid | performance improvement: at least 6.4% | Helped |
| Use DPMD instead of other diffusion-policy online RL baselines | online RL policy on Ant · Ant | performance improvement: 43.4% | Helped |
| Use DPMD instead of other diffusion-policy online RL baselines | online RL policy on Walker2d · Walker2d | performance improvement: 32.1% | Helped |
| Use DPMD instead of other diffusion-policy online RL baselines | online RL policy on HalfCheetah · HalfCheetah | performance improvement: 23.1% | Helped |
| Use 10 or 20 diffusion steps rather than 30 (ablation) | DPMD · Ant-v4 | performance | Helped |
| Use a linear or cosine noise schedule rather than a variance-preserving schedule (ablation) | DPMD · Ant-v4 | performance | Helped |
| Replace uniform sampling with last-policy or perturbed-policy sampling (ablation) | SDAC sampling distribution h_t · MuJoCo tasks | performance | No clear effect |
Surprising findings
- QSM and QVPO fail on some tasks (e.g., QSM on InvertedDoublePendulum, QVPO on Reacher and Pusher), attributed to non-smooth Q functions and inability to handle negative Q-values.
Limitations
- Results are only on MuJoCo locomotion/manipulation tasks; no evaluation on other domains.
- Log-likelihood for SDAC is approximated by the additive Gaussian noise log-probability after batch action selection.
- High variance on some tasks (e.g., DPMD on Swimmer, SDAC on Ant) and the paper notes stability as a future direction.
Experimental setup
Datasets: OpenAI Gym MuJoCo v4 online interaction
Benchmarks: OpenAI Gym MuJoCo v4 (HalfCheetah, Reacher, Humanoid, Pusher, InvertedPendulum, Ant, Hopper, Swimmer, Walker2d, Inverted
Metrics: average return GPU memory training time
Baselines: QSM QVPO DACER DIPO DPPO PPO TD3 SAC
Builds on: denoising diffusion probabilistic models soft actor-critic policy mirror descent denoising score matching
Closest papers by meaning
- Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Score-Based Diffusion Policy Compatible with Reinforcement Learning via Optimal Transport (ICML 2025 · similarity 0.90)
- Diffusion Actor-Critic with Entropy Regulator (NeurIPS 2024 · similarity 0.90)
- Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.90)
- Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching (NeurIPS 2026 · similarity 0.90)
- Diffusion Policies Creating a Trust Region for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.