Research map / Reinforcement learning / Policy gradient and actor-critic

Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Unifies reverse-trajectory (Flow-GRPO-style) and forward-matching (AWM/DiffusionNFT) diffusion-RL losses via path-space importance sampling and a variance-reduced value-gradient form, then proposes a multi-sample KDE value-gradient estimator and scale-bounded weight families evaluated on SD3.5-M and Qwen-Image.

Problem

Diffusion RL algorithms are fragmented: reverse-trajectory methods use discretized likelihood ratios while forward-matching methods train on reward-labeled noised samples, with no unified view or principled design guidance.

Approach

Path-space importance sampling between SDEs with the same diffusion coefficient yields a policy gradient whose Itô-integral noise term equals a deterministic value-gradient term; this unifies Flow-GRPO, AWM and DiffusionNFT as instances of one five-knob design space, and a KDE estimator reuses the rollout group to reduce variance.

Key results

• KDE value-gradient estimator reduces variance by 67.5% relative to the deterministic one-sample estimator, while the stochastic estimator has 13.8x its variance (SD3.5-Medium, PickScore).
• On OCR reward the method converges 2x faster than AWM and 3x faster than DiffusionNFT.
• Ablation: the full recipe reaches 23.52 after 60 GPU hours versus 23.382 for original DiffusionNFT after 160 hours (about 3x faster).

Surprising findings

• Using the one-sample estimator with inherited weighting alone suffers instability and fails, so variance reduction by itself does not accelerate training; the weight reweighting is indispensable.

Limitations

• KDE bandwidth h != 1 introduces bias; h=1 unbiased but abrupt across steps, so practical h is tuned.
• Weight families chosen empirically; the scale-bounded principle does not select the optimal shape.
• Experiments limited to two models and a small set of rewards at 512x512 resolution with CFG disabled.

Experimental setup

Benchmarks: PickScore OCR GenEval HPSv3

Metrics: reward score training time to convergence estimator variance squared bias

Baselines: Flow-GRPO DanceGRPO TempFlow-GRPO GRPO-Guard AWM DiffusionNFT

Builds on: Flow-GRPO AWM DiffusionNFT GRPO Girsanov theorem

Closest papers by meaning

• TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment (NeurIPS 2026 · similarity 0.92)
• Diffusion Spectral Representation for Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
• Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.90)
• Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.90)
• Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data (NeurIPS 2025 · similarity 0.90)