Research map / Large language models / Alignment and preferences
Diffusion Model Alignment Using Direct Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts Direct Preference Optimization to text-to-image diffusion models (Diffusion-DPO), fine-tuning SDXL-base and SD1.5 on Pick-a-Pic v2 human preferences, and also tests AI-feedback labels.
Problem
Text-to-image diffusion models are typically trained without learning from human preferences, and prior diffusion alignment methods (RL-based or reward-gradient-based) fail to generalize stably to open-vocabulary prompts and diverse feedback.
Approach
Reformulates DPO's preference objective for diffusion models by using the ELBO-based denoising error difference between the policy and a frozen reference model as an implicit likelihood proxy, yielding a simple denoising-loss objective on preferred vs. dispreferred images.
Key results
- DPO-SDXL is preferred 70.0% of the time over SDXL-base for general preference on PartiPrompts, and 64.7% on HPSv2 prompts.
- DPO-SDXL (3.5B) beats SDXL base+refiner (6.6B) with a 69% general preference win rate on PartiPrompts.
- Training SD1.5 on AI-feedback PickScore labels raises the PartiPrompts general preference win rate over the base from 59.8% to 63.3%, comparable to human-label training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune SDXL-base with Diffusion-DPO on Pick-a-Pic preferences | SDXL-1.0 base · PartiPrompts | general preference win rate: 70.0% | Helped |
| Fine-tune SDXL-base with Diffusion-DPO on Pick-a-Pic preferences | SDXL-1.0 base · HPSv2 | general preference win rate: 64.7% | Helped |
| Fine-tune SDXL-base with Diffusion-DPO on Pick-a-Pic preferences | SDXL-(base + refiner) · PartiPrompts | general preference win rate: 69% | Helped |
| Fine-tune SDXL with Diffusion-DPO | SDXL image-to-image editing · TEd-Bench | edit preference win rate: 65% preferred; SDXL 24%, with 11% draws | Helped |
| Train SD1.5 with PickScore-ranked preference pairs (ablation) | SD1.5 · PartiPrompts | general preference win rate: 59.8% to 63.3% | Helped |
| Train using Aesthetic Predictor-ranked pairs (ablation) | SD1.5 · Aesthetic appeal and CLIP text-image alignment | CLIP text-image alignment | Hurt |
| Train using CLIP-ranked preference pairs (ablation) | SD1.5 · Text-image alignment | text-image alignment | Helped |
| Replace Diffusion-DPO with SFT on preferred images (ablation) | SD1.5 · PartiPrompts | win rate over base model: 55.5% | Helped |
| Replace Diffusion-DPO with SFT on preferred images (ablation) | SDXL · Pick-a-Pic preferred-image training | performance | Hurt |
Surprising findings
- SFT on preferred Pick-a-Pic images deteriorates SDXL performance at any learning rate, while DPO improves it.
- DDPO public implementations did not yield stable improvement when training on Pick-a-Pic.
Limitations
- Diffusion-DPO is an offline algorithm; online learning not explored.
- Preferences are not universal; personal or group preference tuning left to future work.
- Data quality matters: SFT and DPO outcomes depend on whether the training data is higher quality than the base model.
Experimental setup
Datasets: Pick-a-Pic v2
Benchmarks: PartiPrompts HPSv2 benchmark prompts TEdBench Pick-a-Pic v2 validation
Metrics: win rate (human majority vote) PickScore HPSv2 reward preference accuracy
Baselines: SDXL-1.0 base SDXL-1.0 base+refiner SD1.5 Emu DDPO DRaFT AlignProp DOODL
Builds on: Direct Preference Optimization Denoising Diffusion Probabilistic Models DDPO DPOK Bradley-Terry model
Closest papers by meaning
- InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignment (CVPR 2025 · similarity 0.94)
- Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences (ICML 2025 · similarity 0.93)
- Aligning Diffusion Models by Optimizing Human Utility (NeurIPS 2024 · similarity 0.93)
- Personalized Preference Fine-tuning of Diffusion Models (CVPR 2025 · similarity 0.93)
- Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback (NeurIPS 2025 · similarity 0.92)
- DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Scheduling (CVPR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.