Research map / Image generation / Text-to-image diffusion
Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes B2-DiffuRL, an RL fine-tuning framework for text-to-image diffusion that uses backward progressive training and branch-based sampling to mitigate sparse rewards, improving prompt-image alignment while preserving diversity.
Problem
RL fine-tuning of text-to-image diffusion models for prompt-image alignment suffers from sparse rewards (only available at the end of denoising), which hampers learning of effective denoising policies and causes loss of image diversity.
Approach
Train the denoising policy backward in time, starting from the last timesteps and progressively extending the training interval to earlier timesteps, and compare best/worst samples within branches that share the same intermediate state so that only the actions in the current interval are credited.
Key results
- On template 1, the CLIPScore curves of B2-DiffuRL are consistently above DDPO across 25k reward queries, with BS adding further gains on top of BPT.
- Inception Score on template 1 after fine-tuning: DDPO 1.2886 vs. ours 1.3127 (original SD 1.3179), showing smaller diversity loss.
- Held-out prompt CLIPScore on template 1: SD 0.3515, DDPO 0.3698, ours 0.3748; human raters preferred ours in the majority of comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use B2-DiffuRL instead of DDPO | Stable Diffusion v1.4 · Template 1: animal activity | IS: 1.2886 -> 1.3127 | Helped |
| Use B2-DiffuRL instead of DDPO | Stable Diffusion v1.4 · Template 2: color fruit/vegetable | IS: 1.3323 -> 1.3579 | Helped |
| Use B2-DiffuRL instead of DDPO | Stable Diffusion v1.4 · Template 3: object predicate object | IS: 1.3273 -> 1.3348 | Helped |
| Use B2-DiffuRL instead of DDPO | Stable Diffusion v1.4 · Template 1: animal activity (unseen prompts) | CLIPScore: 0.3698 -> 0.3748 | Helped |
| Use B2-DiffuRL instead of DDPO | Stable Diffusion v1.4 · Template 2: color fruit/vegetable (unseen prompts) | CLIPScore: 0.3175 -> 0.3252 | Helped |
| Use B2-DiffuRL instead of DDPO | Stable Diffusion v1.4 · Template 3: object predicate object (unseen prompts) | CLIPScore: 0.3134 -> 0.3183 | Helped |
| Add backward progressive training (ablation) | DDPO fine-tuning of Stable Diffusion · Template 1: animal activity | CLIPScore | Helped |
| Add backward progressive training (ablation) | DDPO fine-tuning of Stable Diffusion · Template 1: animal activity | BERTScore | Helped |
| Add branch-based sampling to backward progressive training (ablation) | Stable Diffusion fine-tuning · Template 1: animal activity | CLIPScore | Helped |
| Add branch-based sampling to backward progressive training (ablation) | Stable Diffusion fine-tuning · Template 1: animal activity | BERTScore | Helped |
| Add branch-based sampling to backward progressive training (ablation) | Stable Diffusion fine-tuning · Template 1: animal activity | IS | Helped |
Surprising findings
- Naive RL (DDPO) fine-tuning collapses diverse prompts to a single cartoon style, trading diversity for alignment.
Limitations
- Evaluated only on Stable Diffusion v1.4 with LoRA and a small set of prompt templates.
- Relies on reward models (CLIPScore, BERTScore) whose instability is acknowledged; BERTScore found unstable.
Experimental setup
Datasets: Visual Relation Dataset annotations (template 3 prompts) GPT-4-generated color-fruit/vegetable prompts (template 2) DDPO animal/activity prompt template
Benchmarks: CLIPScore on training-template prompts BERTScore on training-template prompts Inception Score on generated images held-out prompts from the three templates (CLIPScore)
Metrics: CLIPScore BERTScore Inception Score human preference rate
Baselines: DDPO Stable Diffusion v1.4 DPOK policy gradient DPO
Builds on: DDPO DPOK DPO (Diffusion-DPO) policy gradient
Closest papers by meaning
- Training Diffusion Models with Reinforcement Learning (ICLR 2024 · similarity 0.93)
- David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training (ICML 2025 · similarity 0.92)
- GARDO: Reinforcing Diffusion Models without Reward Hacking (NeurIPS 2026 · similarity 0.92)
- Training-Free Diffusion Model Alignment with Sampling Demons (ICLR 2025 · similarity 0.91)
- Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL (NeurIPS 2026 · similarity 0.91)
- Inference-Time Alignment of Diffusion Models with Direct Noise Optimization (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.