Research map / Reinforcement learning / Offline RL
Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an entropy-regularized diffusion policy built on a mean-reverting SDE with tractable entropy, combined with a Q-ensemble lower-confidence-bound for pessimistic value estimation, evaluated on D4RL offline RL.
Problem
Diffusion policies in offline RL overestimate Q-values on out-of-distribution actions, and entropy regularization (useful for exploration) is hard to apply because diffusion policy log-probabilities are intractable and offline entropy can worsen distribution shift.
Approach
A mean-reverting SDE (zero-mean, sigma^2=2theta) transfers actions to a standard Gaussian, giving an optimal posterior sampler and a tractable entropy approximation for the diffusion policy; the policy loss adds this entropy term, and pessimistic LCB of a 64-member Q-ensemble replaces the mean Q-value.
Key results
- Average D4RL AntMaze normalized score of 86.7 vs 69.6 for Diffusion-QL across six AntMaze tasks (5 seeds).
- Average Gym score 89.9 vs 88.0 for Diffusion-QL; Kitchen average 75.7 vs 69.0 for Diffusion-QL.
- Ensemble size 64 gives 85.2 average on selected AntMaze tasks versus 63.3 at M=2 and 74.0 at M=4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add entropy regularization (alpha 0.01) instead of alpha 0 (ablation) | Entropy-regularized diffusion policy · Antmaze-medium-play-v0 | normalized score: 85.7 -> 91.4 | Helped |
| Add entropy regularization (alpha 0.01) instead of alpha 0 (ablation) | Entropy-regularized diffusion policy · HalfCheetah-Medium-v2 | normalized score: 53.7 -> 54.9 | Helped |
| Use automatic entropy-temperature adjustment instead of alpha 0.01 (ablation) | Entropy-regularized diffusion policy · Pen-human-v1 | normalized score: 67.2 -> 78.5 | Helped |
| Increase Q-ensemble size from 2 to 64 (ablation) | Entropy-based diffusion policy · Antmaze-large-play-v0 | normalized score: 54.2 -> 81.2 | Helped |
| Increase Q-ensemble size from 2 to 64 (ablation) | Entropy-based diffusion policy · Antmaze-medium-play-v0 | normalized score: 84.0 -> 91.4 | Helped |
| Increase LCB coefficient beta from 1 to 4 (ablation) | Q-ensemble diffusion policy · Antmaze-medium-play-v0 | normalized score: 82.4 ± 4.9 -> 91.6± 2.3 | Helped |
| Increase diffusion steps from 3 to 5 (ablation) | Entropy-regularized diffusion policy · Halfcheetah-medium-replay-v2 | normalized score: 43.4 -> 57.0 | Helped |
| Increase diffusion steps from 5 to 10 (ablation) | Entropy-regularized diffusion policy · Kitchen-complete-v0 | normalized score: 82.3 -> 92.7 | Helped |
| Increase diffusion steps from 5 to 10 (ablation) | Entropy-regularized diffusion policy · Antmaze-medium-play-v0 | normalized score: 91.6 -> 90.2 | Hurt |
| Add max Q-backup (ablation) | Q-ensemble diffusion policy · Antmaze-large-play-v0 | normalized score: 22.3 ± 7.1 -> 81.2 ± 3.0 | Helped |
| Add max Q-backup (ablation) | Q-ensemble diffusion policy · Antmaze-medium-play-v0 | normalized score: 89.2 ± 2.9 -> 91.6± 2.3 | Helped |
Surprising findings
- Fixed entropy temperature alpha=0.01 hurts slightly on Adroit tasks where precise control matters, and the paper notes random actions can harm precise-control tasks.
- Diffusion-QL performance drops as training proceeds in sparse-reward AntMaze tasks, indicating training instability.
Limitations
- Diffusion policy inference is slower than Gaussian policies (about 4.8s vs 1.45s eval per 1000 steps on AntMaze), limiting use on compute- and power-constrained devices.
- Fixed entropy coefficient can be detrimental in precise-control tasks such as Adroit; tuning alpha is needed for best results.
- Many hyperparameters differ by domain (max Q-backup, loss type, T), reducing the consistency claim.
Experimental setup
Datasets: D4RL Gym (halfcheetah, hopper, walker2d) D4RL AntMaze D4RL Adroit (pen) D4RL Kitchen
Benchmarks: D4RL Gym D4RL AntMaze D4RL Adroit D4RL Kitchen
Metrics: normalized score
Baselines: BC DT CQL IQL IDQL IQL+EDP Diffusion-QL MSG BCQ BEAR
Builds on: Diffusion-QL IR-SDE MSG EDAC Soft Actor-Critic
Closest papers by meaning
- Diffusion Policies Creating a Trust Region for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.91)
- Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.91)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.90)
- Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching (NeurIPS 2026 · similarity 0.89)
- Diffusion Actor-Critic with Entropy Regulator (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.