Research map / Image generation / Text-to-image diffusion
Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SmPO-Diffusion, which replaces binary DPO preference labels with reward-model-derived smoothed preference weights and uses few-step DDIM inversion plus one ReNoise step to estimate diffusion trajectories, aligning SD1.5 and SDXL at lower GPU cost.
Problem
DPO for text-to-image diffusion models uses binary preference labels that ignore preference variability, causing over-optimization, and estimates trajectory likelihoods via forward noising, causing objective misalignment and costly training.
Approach
Smooth the binary preference with reward-model-derived soft weights (softmax of normalized PickScore) that scale the DPO margin, and replace forward-noise trajectory estimates with a few-step DDIM inversion followed by a single ReNoise step so the loss is computed on image-correlated latents.
Key results
- On HPDv2, SmPO-SDXL reaches median PickScore 23.62 vs 22.75 for base SDXL and 23.13 for DPO-SDXL, and HPSv2.1 32.53 vs 30.06 for DPO-SDXL.
- SmPO-SDXL training takes about 150.8 A800 GPU hours vs 976.0 for DPO-SDXL, roughly 15.5% of the cost.
- SmPO-SD1.5 achieves 41.3 GPU hours vs 204.8 for DPO-SD1.5 and 1056 for KTO-SD1.5, with a 66.8% HPSv2.1 win-rate over KTO-SD1.5 on HPDv2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SmPO instead of DPO | SDXL · HPDv2 | HPSv2.1: 30.06 -> 32.53 | Helped |
| use SmPO instead of KTO | SD1.5 · HPDv2 | HPSv2.1: 28.28 -> 29.31 | Helped |
| use SmPO instead of Diffusion-DPO | SDXL training · A800 training | GPU hours: $\sim$ 976.0 -> $\sim$ 150.8 | Helped |
| use SmPO instead of Diffusion-DPO | SD1.5 training · A800 training | GPU hours: $\sim$ 204.8 -> $\sim$ 41.3 | Helped |
| add DDIM Inversion to DPO (ablation) | SD1.5 · HPDv2 ablation | PickScore: 21.29 -> 21.72 | Helped |
| add a ReNoise step after inversion (ablation) | DPO with Inversion, SD1.5 · HPDv2 ablation | PickScore: 21.72 -> 21.87 | Helped |
| add smoothed preference modeling (ablation) | DPO with Inversion and ReNoise, SD1.5 · HPDv2 ablation | PickScore: 21.87 -> 22.08 | Helped |
| increase maximum DDIM inversion steps from 2 to 19 (ablation) | SmPO-SD1.5 · HPDv2 ablation | PickScore: 22.06 -> 22.11 | Helped |
| increase maximum DDIM inversion steps from 2 to 19 (ablation) | SmPO-SD1.5 · HPDv2 ablation | GPU hours: $\sim$ 27.8 -> $\sim$ 67.3 | Hurt |
| vary gamma from 2 through 20 (gamma 10 is the selected setting) (ablation) | SmPO-SD1.5 · HPDv2 ablation | PickScore: 21.71; 21.94; 22.08; 21.96 | Mixed |
| vary beta from 1000 to 5000 (beta 2000 is the selected setting) (ablation) | SmPO-SD1.5 · HPDv2 ablation | PickScore: 21.98; 22.08; 21.95; 21.90 | Mixed |
| vary inversion CFG; CFG 1 gives the best reported PickScore (ablation) | SmPO-SD1.5 · HPDv2 ablation | PickScore: 21.82; 21.89; 21.79; 21.78; 22.08; 21.83; 21.81 | Mixed |
Surprising findings
- Binary-label DPO with forward-noise estimates performs worse than SFT on several metrics for SD1.5, e.g. SFT beats DPO on HPSv2.1 median (28.60 vs 25.11).
- Increasing DDIM inversion steps beyond 9 yields only marginal gains while training cost rises sharply (67.3 vs 41.3 GPU hours at 19 steps).
Limitations
- Relies on Pick-a-Pic v2, which may contain harmful or biased content, e.g. feminized outputs for neutral prompts.
- Smoothed labels are produced by a reward model (PickScore) rather than real human preference distributions, and the choice of reward model changes results.
Experimental setup
Datasets: Pick-a-Pic v2 HPDv2 Parti-Prompts
Benchmarks: HPDv2 Parti-Prompts Pick-a-Pic v2 test set
Metrics: median reward score win-rate GPU hours user study preference
Baselines: SFT Diffusion-DPO Diffusion-KTO MaPO Refiner
Builds on: Diffusion-DPO DDIM inversion Direct Preference Optimization MaPO Diffusion-KTO
Closest papers by meaning
- InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignment (CVPR 2025 · similarity 0.95)
- Diffusion Model Alignment Using Direct Preference Optimization (CVPR 2024 · similarity 0.93)
- Calibrated Multi-Preference Optimization for Aligning Diffusion Models (CVPR 2025 · similarity 0.92)
- Personalized Preference Fine-tuning of Diffusion Models (CVPR 2025 · similarity 0.91)
- Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback (NeurIPS 2025 · similarity 0.91)
- Aligning Diffusion Models by Optimizing Human Utility (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.