Research map / Reinforcement learning / RL from feedback

Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control

NeurIPS 2024 · Reinforcement learning › RL from feedback · approach: Reward modeling versus direct optimization · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Efficient Diffusion Alignment (EDA): pretrains diffusion behavior policies as gradients of scalar "bottleneck" networks, then fine-tunes them with a DPO-style contrastive loss over K sampled actions annotated with Q-values.

Problem

Diffusion policies only expose score functions, not tractable densities, so LLM-style alignment (DPO-like) methods cannot directly fine-tune pretrained diffusion behavior policies to optimize Q-values in offline continuous control.

Approach

Represent the diffusion noise predictor as the negative gradient of a scalar network over actions so the scalar output gives the unnormalized behavior density; then fine-tune the pretrained policy by classifying the optimal action among K candidates, with logits given by the density gap between policy and frozen behavior model, generalizing DPO from pairwise preferences to scalar Q-labels.

Key results

• EDA achieves 83.7 average normalized D4RL score overall, the highest among compared methods (QGPO 80.7, Diffusion-QL 82.1).
• EDA retains about 95% of its locomotion performance using only 1% of Q-labelled data during fine-tuning, outperforming several baselines trained on full data.
• Fine-tuning converges in roughly 20K gradient steps, about 2% of the ~1M pretraining steps.

Measured effects reported in the paper

Surprising findings

• Gaussian policies failed in initial experiments, drastically underperforming even vanilla behavior cloning, highlighting the need for expressive diffusion policies.
• Preference-based (K-way best-action) alignment underperforms the softer value-based objective, with the gap widening as K grows.

Limitations

• Relies on a separately trained IQL critic to annotate Q-values, so alignment quality depends on critic quality.
• Gains reported on D4RL with rejection sampling used in some results; plots intentionally exclude it.
• Hyperparameter beta is tuned per task (sweep over several values), with no single universal setting.

Experimental setup

Datasets: D4RL HalfCheetah D4RL Hopper D4RL Walker2d D4RL AntMaze D4RL Kitchen

Benchmarks: D4RL MuJoCo locomotion D4RL AntMaze D4RL FrankaKitchen

Metrics: normalized D4RL score (mean ± std over 5 seeds) data efficiency fine-tuning convergence steps

Baselines: BCQ CQL IQL DT Decision Diffuser IDQL Diffusion-QL QGPO

Builds on: Direct Preference Optimization Diffusion-QL IDQL Implicit Q-learning VPSDE

Closest papers by meaning

• Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.89)
• Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.89)
• Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.89)
• Forward KL Regularized Preference Optimization for Aligning Diffusion Policies (AAAI 2025 · similarity 0.89)
• Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.89)
• Learning a Diffusion Model Policy from Rewards via Q-Score Matching (ICML 2024 · similarity 0.89)