Research map / Image generation / Text-to-image diffusion
Training Diffusion Models with Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames diffusion denoising as a multi-step MDP and optimizes Stable Diffusion with policy gradient (DDPO) on black-box rewards like compressibility, aesthetic quality, and VLM-judged prompt alignment.
Problem
Diffusion models are trained for likelihood, but downstream objectives (human-perceived quality, prompt alignment, file size) are not likelihoods; existing reward-weighted likelihood methods only approximately optimize such rewards.
Approach
Treat each denoising step as an action in an MDP with exact Gaussian likelihoods, and apply policy gradient estimators (score function and importance-sampled PPO-style clipped updates) to optimize arbitrary black-box rewards, with VLM (LLaVA + BERTScore) feedback replacing human labels.
Key results
- DDPO_IS outperforms RWR-style baselines on compressibility, incompressibility, and aesthetic quality reward curves over reward queries.
- Universal guidance raises aesthetic score from 5.95 to 6.14 on 'wolf' while DDPO_IS reaches 6.63 at 20k reward queries.
- DDPO_IS outperforms DPOK across all four ImageReward prompts without KL regularization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-step policy gradients (DDPO) instead of reward-weighted regression (RWR) | Stable Diffusion v1.4 finetuning · Compressibility, incompressibility, and aesthetic quality | Reward | Helped |
| Use importance sampling (DDPO_IS) instead of the score function estimator (DDPO_SF) (ablation) | DDPO · Compressibility, incompressibility, and aesthetic quality | Reward | Helped |
| Use sparse binary weighting instead of exponentiated-reward weighting (ablation) | RWR · Compressibility, incompressibility, and aesthetic quality | Reward | No clear effect |
| Add classifier-free guidance training (ablation) | RWR_sparse · Repeated rounds of sampling and training | Reward | Helped |
| Increase sampling/training interleaving (ablation) | RWR · Reward optimization | Reward | Mixed |
| Use DDPO_IS instead of DPOK | Stable Diffusion v1-5 with LoRA on four prompts · ImageReward and LAION aesthetic score | Reward | Helped |
| Finetune with DDPO using a VLM-derived prompt-alignment reward | Stable Diffusion text-to-image model · Animal-and-activity prompts | BERTScore recall | Helped |
| Finetune on common-animal prompts for aesthetic quality | Stable Diffusion text-to-image model · Unseen animals | Aesthetic reward | Helped |
| Finetune on common-animal prompts for aesthetic quality | Stable Diffusion text-to-image model · Ordinary objects | Aesthetic reward | Helped |
| Optimize incompressibility beyond the useful checkpoint | DDPO-finetuned Stable Diffusion · Incompressibility | Semantic image content | Hurt |
Surprising findings
- Finetuning on 45 animals and 3 activities generalizes to unseen animals, everyday objects, and the novel activity 'taking an exam'.
- Policy gradient training can degrade into overoptimization, e.g. incompressibility yields pure high-frequency noise and VLM reward is exploited with typographic text.
Limitations
- Overoptimization/reward exploitation with no general prevention; checkpoints chosen manually.
- Evaluation limited to constrained prompt sets (animals performing activities).
- Conditional-only training required CFG training to remain stable across rounds.
Experimental setup
Datasets: ImageNet-1000 category names
Benchmarks: LAION aesthetic predictor score BERTScore prompt alignment JPEG compressibility
Metrics: attained reward vs reward queries ImageReward LAION aesthetic score
Baselines: RWR RWR_sparse DPOK Universal guidance
Builds on: denoising diffusion probabilistic models reward-weighted regression proximal policy optimization classifier-free guidance RLAIF
Closest papers by meaning
- Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards (CVPR 2025 · similarity 0.93)
- Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model (CVPR 2024 · similarity 0.93)
- Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback (NeurIPS 2025 · similarity 0.92)
- Learning to Sample From Diffusion Models via Inverse Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
- D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samples (ICML 2025 · similarity 0.90)
- David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.