Research map / Image generation / Text-to-image diffusion
InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DDIM-InPO, a DPO variant for text-to-image diffusion that reparameterizes DDIM to assign implicit rewards to latent variables and uses DDIM inversion to pick informative latents, fine-tuning SD1.5 and SDXL on Pick-a-Pic v2.
Problem
Aligning text-to-image diffusion models with human preferences via direct preference optimization is inefficient and yields weaker quality because the denoising process is a long Markov chain and reward assignment over all denoising steps is sparse and intractable.
Approach
Treat the diffusion model as a single-step generator by reparameterizing DDIM with an initial-image variable x0(t), then use DDIM inversion to estimate the latent x_t corresponding to each clean image so that only latents strongly tied to the preference data are optimized.
Key results
- InPO-SDXL reaches 90.22% HPSv2 win-rate vs. Base-SDXL... and 65.81% HPSv2 win-rate vs. DPO-SDXL on HPDv2 (reported as 85.38% vs base-SDXL in Table 1).
- InPO-SD1.5 achieves 60.94% HPSv2 win-rate vs. KTO-SD1.5 on HPDv2 and 70.50% PickScore win-rate vs. DPO-SD1.5.
- Training is reported as 3.6x faster than Diffusion-DPO and 18.4x faster than Diffusion-KTO on SD1.5 (57.6 vs 204.8 GPU hours for DPO).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DDIM-InPO instead of Diffusion-DPO | SDXL · HPDv2 test set | HPSv2 win-rate: 65.81% | Helped |
| Use DDIM-InPO instead of Diffusion-KTO | SD1.5 · HPDv2 test set | HPSv2 win-rate: 60.94% | Helped |
| Use DDIM-InPO instead of SFT | SD1.5 · HPDv2 test set | HPSv2 win-rate: 58.91; 58.03 | Mixed |
| Increase β from the default 2000 to 3000, 4000, or 5000 (ablation) | DDIM-InPO on SD1.5 · HPDv2 test set | PickScore median: 21.894; 21.808; 21.796; 21.771 | Mixed |
| Set DDIM inversion steps to n=30 (ablation) | DDIM-InPO on SD1.5 · HPDv2 test set | PickScore median: 21.894 -> 21.957 | Helped |
| Set DDIM inversion steps to n=5 or n=3 (ablation) | DDIM-InPO on SD1.5 · HPDv2 test set | PickScore median: 21.894 -> 21.845; 21.833 | Hurt |
| Use CFG winv=1, 5, or 7.5 during inversion instead of unconditional inversion (ablation) | DDIM-InPO on SD1.5 · HPDv2 test set | Aesthetic median: 5.7734; 5.7381; 5.7320; 5.7495 | Mixed |
| Initialize the model with the SD1.5 base model (ablation) | DDIM-InPO · HPDv2 test set | PickScore median: 20.719 -> 21.763 | Helped |
| Use DDIM-InPO instead of Diffusion-DPO | SD1.5 · HPDv2 test set | GPU hours: 204.8 -> 57.6 | Helped |
Surprising findings
- Classifier-free guidance during DDIM inversion (w_inv > 0) hurts preference fine-tuning, so unconditional inversion is used.
- Initializing from the base model is weaker than the full method, and increasing inversion steps from 10 to 30 improves quality but raises training time.
Limitations
- Offline fine-tuning that is sensitive to dataset quality; harmful or biased content in preference data (e.g., overly feminine images) leaks into outputs.
- Relies on DDIM inversion, which introduces numerical error especially with classifier-free guidance.
Experimental setup
Datasets: Pick-a-Pic v2 HPDv2 Parti-Prompts
Benchmarks: HPDv2 Parti-Prompts
Metrics: win-rate median score training GPU hours user study preference
Baselines: Base-SD1.5 Base-SDXL SFT Diffusion-DPO Diffusion-KTO DRaFT AlignProp DDPO
Builds on: Diffusion-DPO DPO DDIM Diffusion-KTO Pick-a-Pic
Closest papers by meaning
- Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences (ICML 2025 · similarity 0.95)
- Diffusion Model Alignment Using Direct Preference Optimization (CVPR 2024 · similarity 0.94)
- Calibrated Multi-Preference Optimization for Aligning Diffusion Models (CVPR 2025 · similarity 0.92)
- Personalized Preference Fine-tuning of Diffusion Models (CVPR 2025 · similarity 0.91)
- Aligning Diffusion Models by Optimizing Human Utility (NeurIPS 2024 · similarity 0.91)
- Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.