Research map / Reinforcement learning / Policy gradient and actor-critic
Maximum Entropy Reinforcement Learning with Diffusion Policy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses a diffusion model as the policy inside soft actor-critic for maximum-entropy RL, with a Q-weighted noise estimation training target and a numerical-integration log-probability estimate.
Problem
Gaussian policies in SAC are unimodal and explore only one mode in complex multi-goal RL tasks, so they can get stuck in local optima instead of approaching the optimal MaxEnt policy.
Approach
Train a diffusion policy to fit the Boltzmann distribution exp(Q/β) without target samples, using a weighted importance-sampling estimate of the score (Q-weighted Noise Estimation, which avoids Q-gradients), and compute the diffusion policy's log-probability via numerical integration over the noise-prediction error so that the soft Q-function can be used.
Key results
- MaxEntDP surpasses SAC, MEow and TD3 on tasks (a-d) and matches them on (e-f) in MuJoCo learning curves over five runs, with much smaller evaluation variance.
- Q-weighted Noise Estimation yields lower variance of the target score estimate than QSM and iDEM on HalfCheetah-v3, whose learning curves fluctuate after reaching high performance.
- MaxEntDP performs comparably to or better than QSM, DIPO, QVPO and DACER across tasks, without a single competitor winning consistently.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the Gaussian policy with a diffusion policy | MaxEntDP · Mujoco benchmarks | evaluation performance | Helped |
| replace the Gaussian policy with a diffusion policy | MaxEntDP · Mujoco benchmarks | evaluation variance | Helped |
| use Q-weighted Noise Estimation instead of QSM | diffusion policy training · HalfCheetah-v3 | performance stability | Helped |
| use Q-weighted Noise Estimation instead of iDEM | diffusion policy training · HalfCheetah-v3 | performance stability | Helped |
| use Q-weighted Noise Estimation instead of QSM or iDEM | diffusion policy training · HalfCheetah-v3 | score estimation variance | Helped |
| use MaxEntDP rather than competing diffusion-based online RL algorithms | online RL · Mujoco benchmarks | performance | Mixed |
| include policy entropy when computing the Q-function (ablation) | MaxEntDP · Mujoco benchmarks | performance | Helped |
| include policy entropy when computing the Q-function (ablation) | MaxEntDP · Mujoco benchmarks | training variance | Helped |
Limitations
- Temperature coefficient is tuned per task and kept fixed during training.
- Action selection at inference requires sampling M candidates and picking the highest-Q action.
Experimental setup
Benchmarks: MuJoCo 2-D multi-goal point-mass environment
Metrics: episode return score-estimation standard deviation
Baselines: SAC MEow TD3 QSM DIPO QVPO DACER iDEM
Builds on: soft actor-critic diffusion models QSM iDEM MEow
Closest papers by meaning
- DIME: Diffusion-Based Maximum Entropy Reinforcement Learning (ICML 2025 · similarity 0.94)
- Generative Actor-Critic with Soft Bridge Policies (NeurIPS 2026 · similarity 0.92)
- Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching (NeurIPS 2026 · similarity 0.91)
- Diffusion Actor-Critic with Entropy Regulator (NeurIPS 2024 · similarity 0.91)
- S$2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic (ICLR 2024 · similarity 0.90)
- When Maximum Entropy Misleads Policy Optimization (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.