Research map / Reinforcement learning / RL from feedback

GDMD: Guiding Distribution Matching Distillation with Gradient-Based Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL fine-tuning of generative models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GDMD, which uses DMD gradients (as implicit target tensors) rather than raw 4-step samples as the reward signal for RL-based joint distillation of diffusion models, improving few-step SDXL and SD3 generators.

Problem

Joint Distribution Matching Distillation and RL for few-step diffusion models suffers from unreliable sample-based rewards on noisy early-stage outputs and gradient conflicts between RL and DMD objectives, requiring cold-start stages.

Approach

Treat the DMD update target x_tar = G(z-) + (s_real - s_fake) as the object to score with existing reward models, converting the DMD gradient into an implicit target image; scores then weight a DiffusionNFT-style positive/negative preference loss, with gradient collection done from DMD's own randomness rather than SDE sampling.

Key results

• On SD3-Medium, GDMD 4-step reaches HP 0.3076 and ImageReward 1.2702 vs. DMDR* 0.2931 and 1.0120.
• On GenEval, GDMD (4 NFE) reaches overall 0.71 vs. 0.64 for DMDR and 0.62 for the multi-step SDXL teacher with CFG.
• Human study: GDMD preferred over the teacher by 55.1% and over DMDR by 62.7% in image quality.

Surprising findings

• 4-step GDMD students outperform their 50-step multi-step teacher (with CFG) on the reported metrics.
• Training only on CLIP and HPS rewards improves unseen metrics such as Aesthetic Score, Pick Score, and ImageReward.

Limitations

• Gradient-based scoring relies on the DMD fake-score estimator; the DMD gradient only reflects the teacher distribution and the reward signal is indirect.
• Evaluation is limited to two base models (SDXL, SD3-Medium) and CLIP/HPS training rewards.
• Compute, training iterations, and data sizes are not reported in the provided text.

Experimental setup

Datasets: text-to-image-2M ShareGPT-4o-Image

Benchmarks: GenEval ShareGPT-4o-Image prompts (CLIP Score, HP Score, Aesthetic Score, Pick Score, ImageReward)

Metrics: CLIP Score HP Score Aesthetic Score Pick Score ImageReward GenEval overall human preference win rate

Baselines: SDXL-Lightning LCM DMD DMD2 DMDR Hyper-SD Flash Diffusion TDM

Builds on: Distribution Matching Distillation DMDR DiffusionNFT DMD2 Flow-GRPO

Closest papers by meaning

• Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.87)
• TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment (NeurIPS 2026 · similarity 0.86)
• Distribution Backtracking Builds A Faster Convergence Trajectory for Diffusion Distillation (ICLR 2025 · similarity 0.86)
• Offline Behavior Distillation (NeurIPS 2024 · similarity 0.86)
• Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL (NeurIPS 2026 · similarity 0.86)
• Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models (NeurIPS 2026 · similarity 0.86)