Research map / Reinforcement learning / Offline RL
Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Diffusion Actor-Critic (DAC), which solves KL-constrained policy iteration by training a diffusion policy via noise regression with a soft Q-guidance term and an LCB Q-ensemble critic, evaluated on D4RL.
Problem
Offline RL must avoid overestimating values of out-of-distribution actions; policy-regularized methods are hard to apply when the behavior policy is an expressive diffusion model, since density values are unavailable.
Approach
Derive from the closed-form KL-constrained policy improvement solution a diffusion noise regression target with a soft Q-guidance term (Q-gradient weighted by noise scale), so the diffusion model itself is the target policy trained by supervised regression without backpropagating through the denoising chain; critic uses LCB over a Q-ensemble.
Key results
- DAC reports locomotion total 836.4 vs 798.3 for the best prior (DTQL), with medium-tasks gains over 10% (e.g., walker2d-medium 96.8 vs 89.4).
- On antmaze DAC scores 459.9 total, competitive with IDQL-A (474.6) and AlignIQL (474.8); it is weaker on antmaze-large tasks.
- Ablation: soft Q-guidance outperforms hard and denoised Q-guidance across walker2d, hopper, and halfcheetah datasets, e.g., hopper-medium 101.2 vs 103.1 (hard) and 17.8 (denoised).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use soft Q-guidance with noise-scale weighting rather than hard Q-guidance (ablation) | DAC · D4RL locomotion tasks | normalized score: walker2d m: 85.2 ± 16.1 -> 96.8 ± 3.6; hopper m: 103.1 ± 0.2 -> 101.2 ± 2.0; halfcheetah m: 59.5 ± 0.5 -> 59.1 ± 0.4 | Mixed |
| Apply Q-guidance to denoised actions instead of using noise-scaled soft guidance (ablation) | DAC · D4RL locomotion tasks | normalized score: walker2d m: 8.4 ± 2.2 -> 96.8 ± 3.6; hopper m: 17.8 ± 8.2 -> 101.2 ± 2.0; halfcheetah m: 71.9 ± 1.9 -> 59.1 ± 0.4 | Mixed |
| Replace the LCB value target with the ensemble minimum (ablation) | DAC · D4RL locomotion tasks | normalized score: halfcheetah m-e: 43.2 ± 0.1 -> 99.1 ± 0.9 | Hurt |
| Increase Q-ensemble size from two to at least five (ablation) | DAC · hopper tasks | normalized score | Helped |
| Increase evaluation action samples beyond 10 (ablation) | DAC · hopper tasks | normalized score | No clear effect |
| Increase diffusion steps from 5 to 10 or 20 (ablation) | DAC · hopper tasks | normalized score: hopper-medium-v2: 101.2 -> 102.0 -> 99.7; hopper-medium-replay-v2: 103.1 -> 102.3 -> 102.2; hopper-medium-expert-v2: 111 | No clear effect |
| Use DAC instead of the strongest prior baseline | D4RL locomotion benchmarks · locomotion total | normalized score: 798.3 -> 836.4 | Helped |
| Use DAC instead of the strongest prior baseline | D4RL antmaze benchmarks · antmaze total | normalized score: 474.8 -> 459.9 | Hurt |
Surprising findings
- Denoised Q-guidance (backpropagating through denoising, as in Diffusion QL) often fails to generate in-distribution actions and frequently collapses.
- Larger diffusion step counts converged more slowly, contrary to the claim in Diffusion Q-learning; T=5 was used.
Limitations
- Antmaze-large performance lags IDQL-A, attributed to not tuning reward offsets for sparse rewards.
- Hyperparameters (eta, b, rho) are tuned per task within ranges, and the reported results use the final-50k-step average, with some sensitivity to rho.
Experimental setup
Datasets: D4RL halfcheetah D4RL hopper D4RL walker2d D4RL antmaze D4RL adroit D4RL kitchen
Benchmarks: D4RL locomotion D4RL antmaze D4RL adroit D4RL kitchen
Metrics: normalized score
Baselines: Onestep-RL CQL IQL IVR EQL Diffuser DTQL AlignIQL SfBC DQL IDQL-A
Builds on: Diffusion Q-learning IDQL TD3+BC Diffusion models (DDPM) jaxrl
Closest papers by meaning
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.93)
- Learning a Diffusion Model Policy from Rewards via Q-Score Matching (ICML 2024 · similarity 0.92)
- Diffusion Policies Creating a Trust Region for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.92)
- Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.92)
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Diffusion Actor-Critic with Entropy Regulator (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.