Research map / Image generation / Text-to-image diffusion
Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes step-by-step preference optimization (SPO), an online DPO-style method that picks win/lose pairs from a shared denoising latent at each step using a timestep-aware preference model trained on generic Pick-a-Pic preferences, to improve aesthetics of Stable Diffusion models.
Problem
Generic human preference data (e.g., Pick-a-Pic) mixes layout and prompt-alignment judgments with aesthetics, and two-trajectory DPO methods propagate trajectory-level labels to intermediate steps where image pairs differ mostly in layout, making aesthetic alignment of diffusion models hard.
Approach
Discard trajectory-level preference propagation: at each early denoising step, sample k candidates from a shared latent, use a step-aware CLIP-based preference model to pick the best and worst as a win-lose pair, then randomly pick one candidate to continue denoising, so pairs differ only in fine details.
Key results
- On SDXL, SPO reaches PickScore 23.06, HPSV2 31.80, ImageReward 1.0803, Aesthetic 6.364, versus 21.95/26.95/0.5380/5.950 for vanilla SDXL and 22.64/29.31/0.9436/6.015 for Diffusion-DPO.
- In user study, SPO wins 58.27% of visual appeal comparisons against Diffusion-DPO, while GenEval overall for SPO (55.20) is below Diffusion-DPO (59.58).
- Total training GPU hours for SPO are about 20.8% of Diffusion-DPO for SD-1.5 and 4.9% for SDXL (4x A100 used).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SPO instead of Diff.-DPO | SDXL · Pick-a-Pic val. | Aesthetic score: 6.015 -> 6.364 | Helped |
| use SPO instead of Diff.-DPO | SD-1.5 · Pick-a-Pic val. | Aesthetic score: 5.505 -> 5.887 | Helped |
| use SPO instead of Diff.-DPO | SDXL · GenEval | Overall: 59.58 -> 55.20 | Hurt |
| remove timestep conditioning from SPM (ablation) | SPO on SD-1.5 · Pick-a-Pic val. | Aesthetic score: 5.887 -> 5.678 | Hurt |
| replace SPM with PickScore (ablation) | SPO on SD-1.5 · Pick-a-Pic val. | Aesthetic score: 5.887 -> 5.410 | Hurt |
| initialize the next iteration with the win sample instead of random selection (ablation) | SPO on SD-1.5 · Pick-a-Pic val. | Aesthetic score: 5.887 -> 3.963 | Hurt |
| initialize the next iteration with the lose sample instead of random selection (ablation) | SPO on SD-1.5 · Pick-a-Pic val. | Aesthetic score: 5.887 -> 5.338 | Hurt |
| use 2 candidates per denoising step instead of 4 (ablation) | SPO on SD-1.5 · Pick-a-Pic val. | Aesthetic score: 5.887 -> 5.714 | Hurt |
| use 8 candidates per denoising step instead of 4 (ablation) | SPO on SD-1.5 · Pick-a-Pic val. | Aesthetic score: 5.887 -> 5.605 | Hurt |
| vary the number of MSPO inner steps from 4 to 1, 2, 3, 5, or 6 (ablation) | MSPO on SDXL · Pick-a-Pic val. | Aesthetic score: 6.364; 6.359; 6.268; 6.380; 6.423; 6.390 | Mixed |
| apply SPO to timestep range [0-1000] instead of [0-750] (ablation) | SPO on SD-1.5 · Pick-a-Pic val. | Aesthetic score: 5.887 -> 5.111 | Hurt |
| apply SPO only to timestep range [0-250] instead of [0-750] (ablation) | SPO on SD-1.5 · Pick-a-Pic val. | Aesthetic score: 5.887 -> 5.413 | Hurt |
Surprising findings
- SPO's GenEval overall score (55.20) is lower than Diffusion-DPO (59.58), indicating weaker layout/prompt alignment despite better aesthetics.
- Using random selection to initialize the next step outperforms feeding the win or lose sample forward, and the best/worst pair choice is only marginally better than random pair selection.
Limitations
- Not applicable to flow matching models (SD3, Flux) because it requires stochastic intermediate steps.
- Offers limited improvement in image-text alignment and does not capture layout changes well (GenEval below Diffusion-DPO).
- Learns only from crowd-sourced generic preferences without addressing subjective, political or historical aspects of aesthetics.
Experimental setup
Datasets: Pick-a-Pic V1 Pick-a-Pic V2 GenEval PartiPrompts HPSV2 benchmark
Benchmarks: Pick-a-Pic validation unique split (500 prompts) GenEval PartiPrompts HPSV2 benchmark
Metrics: PickScore HPSV2 ImageReward Aesthetic score GenEval overall human win rate
Baselines: Diffusion-DPO MAPO D3PO DDPO Vanilla SDXL Vanilla SD-1.5
Builds on: Diffusion-DPO DPO D3PO PickScore DDPO
Closest papers by meaning
- Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences (ICML 2025 · similarity 0.90)
- Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization (NeurIPS 2025 · similarity 0.89)
- Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback (NeurIPS 2025 · similarity 0.89)
- Personalized Image Editing in Text-to-Image Diffusion Models via Collaborative Direct Preference Optimization (NeurIPS 2025 · similarity 0.89)
- Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization (ICLR 2025 · similarity 0.89)
- DrPO: Drifting Preference Optimization for One-Step Generative Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.