Research map / Reinforcement learning / Policy gradient and actor-critic
Learning a Diffusion Model Policy from Rewards via Q-Score Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Q-score matching (QSM), which trains a diffusion policy's denoising score to match the action gradient of a learned Q-function, enabling off-policy actor-critic RL that differentiates only through the denoiser.
Problem
Training diffusion model policies in off-policy actor-critic RL is hard because standard score matching needs samples from the optimal policy, and prior policy-gradient or Diffusion-QL approaches are sample-inefficient or require backpropagating through the full diffusion sampling chain.
Approach
Theoretically shows that optimizing a policy's score (drift) to align with the action gradient of its Q-function increases expected return, and turns this into a regression loss that matches the denoising model's output to alpha times grad_a Q, so gradients flow only through the denoiser.
Key results
- QSM matches or sometimes outperforms TD3 and SAC across eight DeepMind control tasks, particularly in samples needed to reach high rewards.
- Policy-gradient updates of the same diffusion policy are reported as notably worse than QSM on select continuous control tasks.
- QSM-trained policies show multi-modal action distributions, whereas Diffusion-QL-trained policies with identical architecture do not.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Q-score matching instead of SAC | Diffusion-model policies · DeepMind Control continuous-control tasks | return | Mixed |
| Use Q-score matching instead of TD3 | Diffusion-model policies · DeepMind Control continuous-control tasks | return | Mixed |
| Update the diffusion policy with policy gradients instead of Q-score matching | Diffusion-model policies · Select continuous-control tasks | return | Hurt |
| Train the denoising submodel with Q-score matching instead of Diffusion-QL | Diffusion-model policies · Continuous-control tasks | return | Mixed |
| Train the denoising submodel with Q-score matching instead of Diffusion-QL | Diffusion-model policies · Continuous-control tasks | policy-gradient computation time | Helped |
| Increase the number of diffusion sampling steps | Q-score matching · Walker-walk | training stability | No clear effect |
| Increase the number of diffusion sampling steps | Diffusion-QL · Walker-walk | training stability | Hurt |
Surprising findings
- Policy gradient updates for diffusion policies fail to learn optimal policies in the off-policy setting, performing worse than QSM.
Limitations
- Theory assumes a fixed noise model Sigma_a and a continuous-time idealization; the discrete algorithm is an approximation.
- Exploration is handled only by adding small Gaussian noise; other exploration strategies are left to future work.
- Experiments are limited to simulated DeepMind control tasks, with no real-robot evaluation.
Experimental setup
Datasets: DeepMind Control Suite
Benchmarks: DeepMind Control Suite
Metrics: episode return sample efficiency
Baselines: TD3 SAC Diffusion-QL policy gradient for diffusion policies
Builds on: Diffusion-QL DDPG TD3 SAC soft policy iteration
Closest papers by meaning
- Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.92)
- Diffusion Policies Creating a Trust Region for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.90)
- Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control (NeurIPS 2025 · similarity 0.90)
- Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.90)
- Efficient Online Reinforcement Learning for Diffusion Policy (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.