Research map / Image generation / Text-to-image diffusion
Aligning Diffusion Models by Optimizing Human Utility
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Diffusion-KTO adapts the Kahneman-Tversky utility-maximization objective to text-to-image diffusion models so they can be aligned using only per-image binary (like/dislike) feedback instead of pairwise preferences or a reward model.
Problem
Aligning text-to-image diffusion models with human preferences typically requires pairwise preference data or a reward model with RL, and backpropagating through the sampling chain is memory-expensive; per-image binary feedback is abundant but unused.
Approach
Extends KTO's prospect-theory utility objective to diffusion models by applying a per-denoising-step implicit reward (beta log ratio of step-wise policy vs reference) with a sigmoid utility, signed by desirability, and a batch-estimated KL reference point, so training uses unpaired desirable/undesirable images.
Key results
- Diffusion-KTO (SD v1-5) achieves automatic win-rates up to 87.2% (ImageReward) against SD v1-5 and 86.0% on Aesthetic on Pick-a-Pic prompts.
- Against Diffusion-DPO, Diffusion-KTO wins 74.2% on LAION Aesthetic and 61.8% on PickScore on Pick-a-Pic prompts.
- Human judges prefer Diffusion-KTO over SD v1-5 at a 75% win-rate and over Diffusion-DPO at 69%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Align with per-image binary feedback using Diffusion-KTO rather than Diffusion-DPO | Stable Diffusion v1-5 · Pick-a-Pic v2 test | Aesthetic win rate: 74.2 | Helped |
| Align with per-image binary feedback using Diffusion-KTO rather than Diffusion-DPO | Stable Diffusion v1-5 · Pick-a-Pic v2 test | ImageReward win rate: 78.4 | Helped |
| Align with per-image binary feedback using Diffusion-KTO rather than Diffusion-DPO | Stable Diffusion v1-5 · HPS v2 test | CLIP win rate: 47.3±1.96 | Hurt |
| Align with per-image binary feedback using Diffusion-KTO rather than Diffusion-DPO | Stable Diffusion v1-5 · PartiPrompts | HPS v2 win rate: 46.1±1.95 | Hurt |
| Use KTO utility instead of Loss-Averse utility (ablation) | Diffusion-KTO · HPS v2 test | PickScore | Helped |
| Use KTO utility instead of Risk-Seeking utility (ablation) | Diffusion-KTO · HPS v2 test | PickScore | Helped |
| Label images desirable if preferred at least once, rather than only if preferred in every comparison (ablation) | Diffusion-KTO · HPS v2 test | PickScore | Helped |
| Sample 80% of each minibatch from desirable images (ablation) | Diffusion-KTO · HPS v2 test | PickScore | Helped |
| Increase β toward 5000 (ablation) | Diffusion-KTO · HPS v2 test | PickScore | Helped |
Surprising findings
- The Kahneman-Tversky sigmoid utility outperformed loss-averse and risk-seeking utilities in ablations, while the loss-averse utility reduces to a Diffusion-DPO-like objective.
- Training on Pick-a-Pic increases NSFW rate marginally (5.4% vs 4.4% for SD v1-5 on Pick-a-Pic prompts).
Limitations
- Trained on Pick-a-Pic, whose preference distribution may be skewed toward inappropriate or unwanted imagery and biased artistic styles.
- The choice of utility function and its underlying human-behavior assumptions remains an open question; only three utilities were examined.
- Inherits weaknesses of the pretrained T2I model, including negative stereotypes.
Experimental setup
Datasets: Pick-a-Pic v2 HPS v2 PartiPrompts
Benchmarks: Pick-a-Pic v2 test set HPS v2 test prompts PartiPrompts
Metrics: win-rate LAION aesthetics PickScore ImageReward CLIP score HPS v2 human win-rate
Baselines: Stable Diffusion v1-5 SFT CSFT AlignProp D3PO Diffusion-DPO
Builds on: KTO Diffusion-DPO D3PO DPO Prospect theory
Closest papers by meaning
- Diffusion Model Alignment Using Direct Preference Optimization (CVPR 2024 · similarity 0.93)
- Calibrated Multi-Preference Optimization for Aligning Diffusion Models (CVPR 2025 · similarity 0.93)
- Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization (NeurIPS 2025 · similarity 0.92)
- A Dense Reward View on Aligning Text-to-Image Diffusion with Preference (ICML 2024 · similarity 0.92)
- Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models (ICLR 2025 · similarity 0.91)
- Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.