Research map / Reinforcement learning / Policy gradient and actor-critic

Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient

NeurIPS 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Policy gradients for special settings · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DDiffPG, an actor-critic method that trains diffusion policies from scratch for multimodal behaviors by combining novelty-based exploration, hierarchical trajectory clustering, mode-specific Q-functions, and mode-embedding conditioning.

Problem

Online RL policies are typically unimodal (deterministic or Gaussian), and standard RL greediness collapses policies onto a single behavioral mode; diffusion policies are hard to train online because policy likelihood is intractable and Q-gradients vanish through the denoising chain.

Approach

Diffusion policy gradient converts action-gradient ascent into target actions stored in the replay buffer and trains the diffusion model with an MSE/BC-style loss; multimodality is explicitly discovered by clustering goal-reaching trajectories, with a separate Q-function per mode and multimodal batches to prevent mode collapse.

Key results

• On AntMaze-v3 DDiffPG discovers 4.3 modes on average with 0.98 success rate, versus 1.0 for DIPO and TD3 which are unimodal (1 mode).
• On AntMaze-v1 DDiffPG finds 2 modes with 1.0 success rate and episode length 75.7, while Diffusion-QL reaches 0.63 success.
• In randomized-obstacle replanning, DDiffPG reaches 1.0 success rate versus 0.5 for TD3 and DIPO, though the paper reports DDiffPG is about five times slower than TD3 and SAC.

Measured effects reported in the paper

Surprising findings

• Diffusion-QL and Consistency-AC often have zero actor gradients throughout training in some seeds, causing high variance, whereas the proposed MSE-to-target-action objective is more stable.
• Single-mode baselines such as DIPO, Diffusion-QL and Consistency-AC quickly collapse to one mode even when the policy is diffusion-parameterized.

Limitations

• Clustering relies on distance over 2D/end-effector positions, which may fail for locomotion gaits with similar position changes.
• Intrinsic motivation may degrade in larger spaces since many viable solutions must be discovered.
• Diffusion models increase training and inference wall-clock time, limiting real-time robotic use.

Experimental setup

Datasets: D4RL AntMaze environments Franka robotic manipulation environments

Benchmarks: AntMaze-v1 AntMaze-v2 AntMaze-v3 AntMaze-v4 Reach Peg-in-hole Drawer-close Cabinet-open

Metrics: success rate episode length number of modes discovered state coverage sample efficiency

Baselines: DIPO Diffusion-QL Consistency-AC RPG TD3 SAC

Builds on: DIPO Diffusion-QL Deterministic policy gradient Random Network Distillation TD3

Closest papers by meaning

• Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.90)
• Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.90)
• Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularization (NeurIPS 2025 · similarity 0.90)
• Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.89)
• Diffusion Actor-Critic with Entropy Regulator (NeurIPS 2024 · similarity 0.89)
• Diffusion Policies Creating a Trust Region for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)