Research map / Image generation / Text-to-image diffusion
Calibrated Multi-Preference Optimization for Aligning Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CaPO, which aligns text-to-image diffusion models using calibrated win-rate rewards from multiple reward models (no human preference data), with frontier-based rejection sampling for multi-reward pair selection and a regression-style diffusion preference loss.
Problem
Aligning text-to-image diffusion models with preferences without costly human-annotated preference data, and handling multiple, inconsistent reward models; pairwise DPO-style methods use raw reward values that are poorly calibrated.
Approach
Calibrate each sample's reward as its average pairwise win-rate against other samples from the pretrained model (approximating expected win-rate over the reference), select training pairs from Pareto frontiers via non-dominated sorting, and fit the diffusion implicit reward difference to the calibrated reward difference with a regression loss.
Key results
- On GenEval, CaPO raises SDXL overall from 0.55 to 0.59 and SD3-M from 0.68 to 0.71.
- In multi-reward SDXL experiments, CaPO with FRS reaches 61.2% MPS win-rate and 79.2% VILA win-rate, versus 58.1% and 78.6% for DPO with FRS.
- Compared with Diffusion-DPO trained on human Pick-a-pic data, CaPO achieves higher Pickscore (22.83 vs 22.71), MPS (11.71 vs 11.59), and VILA (6.141 vs 6.049), while Diffusion-DPO is better on VQAscore (0.834 for both at near parity).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CaPO fine-tuning | SDXL · GenEval | Overall: 0.55 -> 0.59 | Helped |
| Apply CaPO fine-tuning | SD3-M · GenEval | Overall: 0.68 -> 0.71 | Helped |
| Apply synthetic-data CaPO fine-tuning | SDXL · Parti prompts | Pickscore: 22.71 -> 22.83 | Helped |
| Apply synthetic-data CaPO fine-tuning | SDXL · Parti prompts | VQAscore: 0.834 -> 0.838 | Helped |
| Use frontier-based rejection sampling instead of sum-of-rewards pair selection | CaPO · SDXL multi-reward | MPS Win (%): 61.2 vs 61.2 | Helped |
| Use frontier-based rejection sampling instead of sum-of-rewards pair selection | CaPO · SDXL multi-reward | VQAscore Win (%): 52.5 -> 54.6 | Helped |
| Use frontier-based rejection sampling instead of sum-of-rewards pair selection | CaPO · SD3-M multi-reward | VILA Win (%): 57.0 -> 69.3 | Helped |
| Replace CaPO FRS with a rewarded soup of single-reward models | CaPO · SDXL multi-reward | VILA Win (%): 79.2 -> 77.6 | Hurt |
| Add sigmoid loss weighting instead of constant weighting (ablation) | CaPO on SDXL · SDXL multi-reward | MPS Win (%): 56.5 -> 61.2 | Helped |
| Set sigmoid weighting bias to b = 1.5 (ablation) | CaPO on SDXL · SDXL multi-reward | MPS Win (%): 59.1 -> 61.2 | Helped |
| Set sigmoid weighting bias to b = 2.0 rather than b = 1.5 (ablation) | CaPO on SDXL · SDXL multi-reward | MPS Win (%): 61.2 -> 58.6 | Hurt |
| Use top-1 and worst-1 selection from 16 rather than 8 generated images per prompt (ablation) | CaPO · Training data generation | performance | Mixed |
Surprising findings
- DPO on a single reward (VILA) can cause a significant drop in VQAscore, indicating reward hacking, while IPO is more robust but gains less.
- Rewarded soup (model merging) performs comparably to FRS for DPO and IPO objectives in the multi-reward setting.
Limitations
- Offline data generation suffers from slow convergence; online extension left for future work.
- Improvements are bounded by the capability of the original base model (e.g., text rendering for SDXL).
Experimental setup
Datasets: DiffusionDB Pick-a-pic v2 prompts Parti prompts DPG-bench prompts
Benchmarks: GenEval T2I-CompBench Parti prompts win-rate DPG-bench win-rate
Metrics: win-rate average reward score GenEval overall score T2I-CompBench scores
Baselines: DPO IPO Diffusion-DPO rewarded soup FLUX-dev FLUX-schnell SD3.5-L SDXL SD3-M
Builds on: Diffusion-DPO IPO DPO rewarded soups
Closest papers by meaning
- A Dense Reward View on Aligning Text-to-Image Diffusion with Preference (ICML 2024 · similarity 0.93)
- Aligning Diffusion Models by Optimizing Human Utility (NeurIPS 2024 · similarity 0.93)
- InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignment (CVPR 2025 · similarity 0.92)
- Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences (ICML 2025 · similarity 0.92)
- Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization (NeurIPS 2025 · similarity 0.91)
- Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.