Research map / Image generation / Text-to-image diffusion
A Dense Reward View on Aligning Text-to-Image Diffusion with Preference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a dense-reward view of DPO-style T2I preference alignment that adds temporal discounting (gamma<1) to the trajectory-level Bradley-Terry loss, emphasizing early diffusion steps, trained via LoRA on Stable Diffusion v1.5.
Problem
Existing DPO-style explicit-reward-free text-to-image preference alignment treats the whole diffusion reverse chain as a single bandit action, ignoring the sequential generation process, which may hurt efficacy and efficiency.
Approach
Assume a latent per-step reward and introduce a temporal discount factor gamma into the DPO-style Bradley-Terry loss (lower-bounded via reward shaping and a partition-function ordering argument), so earlier reverse steps receive more weight; gamma=1 recovers the trajectory-level DPO variant.
Key results
- On the HPSv2 test set, the method reaches an average HPSv2 score of 30.09 versus 29.04 for the trajectory-level reward variant and 27.86 for the best listed baseline (Dreamlike Photoreal 2.0).
- Human win rate of the method over SD1.5 is 76.8%, over Dreamlike Photoreal 2.0 is 68.3%, and over the trajectory-level reward variant is 65.1%.
- Smaller discount factors (gamma=0.9 or 0.95) train faster and better than gamma=1 in both single and multiple prompt experiments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dense reward with γ=0.9 instead of trajectory-level reward with γ=1 (ablation) | Stable Diffusion v1.5 preference alignment · HPSv2 test, multiple prompts | HPSv2 Average: 29.04 -> 30.09 | Helped |
| Use dense reward with γ=0.9 instead of trajectory-level reward with γ=1 (ablation) | Stable Diffusion v1.5 preference alignment · HPSv2 test, multiple prompts | Aesthetic: 5.94 -> 6.31 | Helped |
| Reduce γ from 1.0 to 0.9 or 0.95 (ablation) | Dense reward preference alignment · Single and multiple prompt experiments | Preference-generating score | Helped |
| Reduce γ from 1.0 to 0.99 (ablation) | Dense reward preference alignment · Single and multiple prompt experiments | Training efficiency and stability | Helped |
| Vary KL coefficient C across 7.5, 10.0, 12.5, 15.0, and 20.0 (ablation) | Dense reward preference alignment · Multiple prompts, HPSv2 test | HPSv2 Average: 29.79 { (0.26) }; 29.99 { (0.23) }; 30.09 { (0.22) }; 29.88 { (0.20) }; 29.93 { (0.20) } | Mixed |
| Set KL coefficient C=12.5 rather than C=7.5, 10.0, 15.0, or 20.0 (ablation) | Dense reward preference alignment · Multiple prompts, HPSv2 test | HPSv2 Average: 29.79 { (0.26) }; 29.99 { (0.23) }; 29.88 { (0.20) }; 29.93 { (0.20) } -> 30.09 { (0.22) } | Helped |
Surprising findings
- Even gamma=0.99, a small break of temporal symmetry, improves training efficiency and stability over gamma=1.
Limitations
- Requires storing full generation reverse chains, adding CPU memory and storage overhead.
- Off-policy iterative collection adds complexity and cost relative to pure offline approaches.
- Multiple-prompt experiments use N_step=1 due to GPU memory constraints.
Experimental setup
Datasets: HPSv2 train prompts HPSv2 test prompts DPOK single prompts
Benchmarks: ImageReward Laion Aesthetic score HPSv2 test set
Metrics: ImageReward Aesthetic score HPSv2 score human win rate
Baselines: DPOK Trajectory-level Reward (gamma=1 DPO-style) Stable Diffusion v1.5 Stable Diffusion v2.0 SDXL Refiner 0.9 Dreamlike Photoreal 2.0 DALL·E 2 SFT
Builds on: DPO Diffusion-DPO DPOK reward shaping Bradley-Terry model
Closest papers by meaning
- Calibrated Multi-Preference Optimization for Aligning Diffusion Models (CVPR 2025 · similarity 0.93)
- Aligning Diffusion Models by Optimizing Human Utility (NeurIPS 2024 · similarity 0.92)
- InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignment (CVPR 2025 · similarity 0.91)
- Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences (ICML 2025 · similarity 0.90)
- Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization (NeurIPS 2025 · similarity 0.90)
- Improving Long-Text Alignment for Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.