Research map / Reinforcement learning / Policy gradient and actor-critic
DIME: Diffusion-Based Maximum Entropy Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIME, a maximum-entropy RL method that trains diffusion policies by optimizing a lower bound on the marginal entropy via approximate inference (denoising diffusion with OU process), with a CrossQ-style critic and distributional Q.
Problem
Maximum entropy RL with expressive diffusion policies is hard because the marginal entropy of a diffusion policy is intractable, so existing diffusion RL methods rely on Gaussian or uniform noise for exploration.
Approach
Derive a lower bound on the marginal entropy of the denoising diffusion process using the time-reversed approximate inference view, cast MaxEnt policy improvement as minimizing KL between the diffusion's joint path distribution and the exp-Q-weighted noising process, and optimize this via reparameterized diffusion sampling without needing samples from the target policy.
Key results
- DIME significantly outperforms other diffusion-based baselines on all evaluated environments.
- DIME outperforms CrossQ and all diffusion baselines on Humanoid-v3 and dog-run, and outperforms BRO on 10 of 13 environments per the paper's summary.
- DIME with 16 diffusion steps trains in about 4.5h versus BRO's reported 8.5h on humanoid-run, with UTD ratio 2 versus BRO's 10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DIME instead of diffusion-based baselines | DIME · 13 continuous high-dimensional control benchmarks | return or success rate | Helped |
| use DIME instead of CrossQ | DIME · Humanoid-v3 | return | Helped |
| use DIME instead of CrossQ | DIME · Ant-v3 | return | No clear effect |
| use DIME instead of BRO | DIME · DMC dog-run | return | Helped |
| use DIME instead of BRO | DIME · DMC humanoid-run and humanoid-walk | return | Hurt |
| use a diffusion policy instead of a Gaussian policy (ablation) | DIME · DMC humanoid-run | aggregated return | Helped |
| use a diffusion policy instead of a Gaussian policy (ablation) | DIME · DMC dog-run | convergence speed | Helped |
| set reward scaling α to 0.1 (ablation) | DIME · DMC dog-run | task return | Hurt |
| set reward scaling α≤10^-5 (ablation) | DIME · DMC dog-run | task return | Hurt |
| increase reward scaling from α=10^-12 toward α=10^-3 (ablation) | DIME · DMC dog-run | end performance | Helped |
| increase diffusion steps from 2 to 4 or 8 (ablation) | DIME · DMC humanoid-run | performance | Helped |
| increase diffusion steps from 4 or 8 to 16 or 32 (ablation) | DIME · DMC humanoid-run | performance | Helped |
Surprising findings
- DIME without distributional Q still performs on par with or better than BRO on the dog tasks.
- Performance saturates from 16 diffusion steps onward, with 2 steps performing badly.
Limitations
- Theoretical policy iteration guarantees assume finite action spaces.
- Only denoising (OU-based) diffusion processes are explored in the main experiments; general bridge variant only preliminary.
- Baseline QSM required hyperparameter tuning, and BRO results are taken from provided learning curves rather than rerun.
Experimental setup
Datasets: MuJoCo Gym Ant-v3 MuJoCo Gym Humanoid-v3 DeepMind Control dog and humanoid tasks MyoSuite hand tasks
Benchmarks: Ant-v3 Humanoid-v3 DMC dog-run/walk/trot/stand DMC humanoid-run/walk/stand MyoSuite object-hold-hard MyoSuite reach-hard MyoSuite key-turn-hard MyoSuite pen-twirl-hard
Metrics: interquartile mean return success rate 95% stratified bootstrap confidence interval
Baselines: QSM Diffusion-QL Consistency-AC DIPO QVPO DACER CrossQ BRO BRO Fast
Builds on: Soft Actor-Critic CrossQ Denoising diffusion models with OU process BRO DACER
Closest papers by meaning
- Maximum Entropy Reinforcement Learning with Diffusion Policy (ICML 2025 · similarity 0.94)
- Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching (NeurIPS 2026 · similarity 0.93)
- Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models (NeurIPS 2024 · similarity 0.92)
- Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.90)
- Diffusion Actor-Critic with Entropy Regulator (NeurIPS 2024 · similarity 0.90)
- When Maximum Entropy Misleads Policy Optimization (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.