Research map / Image generation / Text-to-image diffusion
Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Diffusion-DRO, an inverse-RL max-margin preference learning method for text-to-image diffusion that ranks expert demonstrations against policy samples, yielding a denoising-style loss with a thresholded ranking loss, fine-tuned from SD v1-5.
Problem
Aligning text-to-image diffusion models with human preferences without paired comparison data, avoiding the non-linear sigmoid probability estimation of DPO-style objectives and the limited diversity of offline preference datasets.
Approach
Casts preference learning as a max-margin inverse RL ranking problem between expert demonstrations (top-K by an automated score) and online policy samples, parameterizing the trajectory reward implicitly via a denoising model; a thresholded ranking loss clips the margin once satisfied. No reward model or paired data is needed.
Key results
- Diffusion-DRO achieves win rates above 60% on Aesthetic Score and nearly 70% on ImageReward against SPIN-Diffusion, and over 80% on PickScore, Aesthetic, and ImageReward against Diffusion-SPO w/ SFT (automated win rates).
- In the user study on MTurk, Diffusion-DRO achieves a 75% win rate against SD v1-5 and 56.67% against Diffusion-DPO and 66.67% against Diffusion-KTO.
- Diffusion-DRO (PickScore) reaches 21.76 PickScore and 0.8636 ImageReward on Pick-a-Pic v2 test, above SD v1-5 w/ SFT (21.24, 0.7623).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Diffusion-DRO instead of the strongest reported baseline | SD v1-5 · Pick-a-Pic v2 test | PickScore, HPSv2, Aesthetic, CLIP Score, and ImageReward: 21.76±1.51 vs 21.40±1.38; 28.49±1.75 vs 28.08±1.72; 6.40±.976 vs 6.35±.995; 0.3446±.057 vs 0.3438±.056; 0.8636±.907 vs 0 | Helped |
| Use Diffusion-DRO instead of the strongest reported baseline | SD v1-5 · HPDv2 Benchmark | PickScore, HPSv2, Aesthetic, CLIP Score, and ImageReward: 22.04±1.28 vs 21.66±1.27; 29.05±1.66 vs 28.77±1.59; 6.70±.873 vs 6.65±.855; 0.3560±.053 vs 0.3584±.051; 0.9069±.833 vs 0 | Mixed |
| Fine-tune SD v1-5 with Diffusion-DRO rather than leave it unmodified | SD v1-5 · HPDv2 Benchmark user study | human preference win rate: 75% | Helped |
| Fine-tune SD v1-5 with Diffusion-DRO rather than Diffusion-DPO w/ SFT | SD v1-5 · HPDv2 Benchmark user study | human preference win rate: 59.6% | Helped |
| Fine-tune SD v1-5 with Diffusion-DRO rather than Diffusion-KTO w/ SFT | SD v1-5 · HPDv2 Benchmark user study | human preference win rate: 63.82% | Helped |
| Increase the number of expert demonstrations (ablation) | Diffusion-DRO · Pick-a-Pic v2 test and HPDv2 Benchmark | CLIP Score | Helped |
| Use Diffusion-DRO rather than SFT trained with the same expert demonstrations (ablation) | SD v1-5 · Pick-a-Pic v2 test and HPDv2 Benchmark | PickScore, Aesthetic, and ImageReward | Helped |
Surprising findings
- SFT on expert demonstrations alone is a competitive baseline, and CLIP Score decreases after fine-tuning on expert demonstrations because the text encoder drifts from the CLIP distribution.
- Diffusion-SPO w/ SFT fails to improve over SD v1-5 w/ SFT, since fine-tuning reduces variation in step-aware preference pairs.
Limitations
- Expert demonstrations derived from data may introduce distributional bias (e.g., simpler prompts favored) and non-expert data is discarded.
- Preferences treated as binary rankings, losing list-wise ranking information.
- Training uses automated scores (HPSv2/PickScore) as proxies for expert selection, and evaluation relies largely on automated metrics.
Experimental setup
Datasets: Pick-a-Pic v2 HPDv2 benchmark HPSv2 benchmark
Benchmarks: Pick-a-Pic v2 test HPDv2 benchmark
Metrics: win rate PickScore HPSv2 Aesthetic Score CLIP Score ImageReward
Baselines: SD v1-5 SD v1-5 w/ SFT SPIN-Diffusion Diffusion-SPO Diffusion-DPO Diffusion-KTO
Builds on: Diffusion-DPO DPOK SPIN-Diffusion Diffusion-KTO Diffusion-SPO
Closest papers by meaning
- Training Diffusion Models with Reinforcement Learning (ICLR 2024 · similarity 0.92)
- Diffusion Model Alignment Using Direct Preference Optimization (CVPR 2024 · similarity 0.92)
- Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models (ICLR 2025 · similarity 0.92)
- Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model (CVPR 2024 · similarity 0.92)
- Personalized Preference Fine-tuning of Diffusion Models (CVPR 2025 · similarity 0.91)
- Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.