Research map / Reinforcement learning / Policy gradient and actor-critic
Diffusion Actor-Critic with Entropy Regulator
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DACER, an online actor-critic RL method that uses a diffusion model's reverse process as the policy, estimates its entropy via a Gaussian mixture model fit to sampled actions, and adapts exploration noise with a learned alpha.
Problem
Traditional online RL policies are diagonal Gaussians, which cannot represent multimodal optimal policies; diffusion policies in RL have mostly been restricted to offline RL, and their entropy has no analytic form, complicating maximum-entropy exploration.
Approach
Treat the reverse diffusion chain (DDPM-style) as the policy and optimize it end-to-end by backpropagating Q-value gradients through the whole denoising chain; estimate the policy entropy by fitting a 3-component GMM to 200 sampled actions per state, and learn a scalar alpha that scales the variance of Gaussian noise added to the diffusion output.
Key results
- DACER achieves 11888 ± 244 on Humanoid-v3 versus 10829 for DSAC and 9335 for SAC, and the best mean return on all eight MuJoCo tasks in the table (ties on Inverted Double Pendulum and Pusher).
- On Walker2d-v3, DACER reaches 6701 ± 62 versus 6424 for DSAC.
- Ablation on Walker2d-v3 shows removing the entropy regulator (DAC) substantially lowers performance, and 20 diffusion steps performed best among 10, 20, 30.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DACER instead of DSAC | MuJoCo control · Humanoid-v3 | average final return: 10829 ± 243 -> 11888 ± 244 | Helped |
| Use DACER instead of DSAC | MuJoCo control · Ant-v3 | average final return: 7086 ± 261 -> 9108 ± 103 | Helped |
| Use DACER instead of DSAC | MuJoCo control · HalfCheetah-v3 | average final return: 17025 ± 157 -> 17177 ± 176 | Helped |
| Use DACER instead of DSAC | MuJoCo control · Walker2d-v3 | average final return: 6424 ± 147 -> 6701 ± 62 | Helped |
| Use DACER instead of DSAC | MuJoCo control · InvertedDoublePendulum-v3 | average final return: 9360 ± 0 -> 9360 ± 0 | No clear effect |
| Use DACER instead of DSAC | MuJoCo control · Hopper-v3 | average final return: 3660±533 -> 4104 ± 49 | Helped |
| Use DACER instead of DSAC | MuJoCo control · Pusher-v2 | average final return: -19 ± 1 -> -19 ± 1 | No clear effect |
| Use DACER instead of DSAC | MuJoCo control · Swimmer-v3 | average final return: 138±6 -> 152 ± 7 | Helped |
| Add the entropy regulator to diffusion actor-critic (ablation) | DAC on Walker2d-v3 · Walker2d-v3 | return | Helped |
| Use entropy-based adaptive noise tuning instead of fixed or linear-decay noise (ablation) | DACER · Walker2d-v3 | return | Helped |
| Increase reverse diffusion steps from 20 to 30 (ablation) | DACER · Walker2d-v3 | return | Hurt |
| Vary reverse diffusion steps among 10, 20, and 30 (ablation) | DACER · Walker2d-v3 | return | Mixed |
Surprising findings
- Diffusion policy trained purely by Q-maximization without entropy regulation produces overly deterministic actions and poor performance.
- Increasing diffusion steps to 30 caused training instability and gradient explosion, so more steps did not help.
Limitations
- GMM entropy estimation is slow (about 40 ms) and needs many samples, so entropy is estimated only every 10,000 iterations, which prevents full integration with maximum-entropy RL.
- Evaluation is limited to MuJoCo tasks and one small multi-goal environment.
Experimental setup
Datasets: MuJoCo benchmark tasks (Humanoid-v3, Ant-v3, HalfCheetah-v3, Walker2d-v3, InvertedDoublePendulum-v3, Hopper-v3, Pusher-v Multi-goal 2D point-mass environment
Benchmarks: MuJoCo Multi-goal 2D navigation
Metrics: average final return training return curves
Baselines: DDPG TD3 PPO SAC DSAC TRPO QSM
Builds on: DDPM DSAC TD3 SAC Diffusion-QL
Closest papers by meaning
- Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.91)
- Maximum Entropy Reinforcement Learning with Diffusion Policy (ICML 2025 · similarity 0.91)
- Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching (NeurIPS 2026 · similarity 0.90)
- Efficient Online Reinforcement Learning for Diffusion Policy (ICML 2025 · similarity 0.90)
- DIME: Diffusion-Based Maximum Entropy Reinforcement Learning (ICML 2025 · similarity 0.90)
- Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.