Research map / Image generation / Text-to-image diffusion
CPO: Condition Preference Optimization for Controllable Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Condition Preference Optimization (CPO), which fine-tunes ControlNet-style controllable diffusion models by preferring ground-truth control signals over perturbed ones instead of preferring generated images, claiming lower loss variance than Diffusion-DPO and better controllability than ControlNet++.
Problem
Controllability in text-to-image diffusion models with dense conditions (segmentation, pose, edges, depth) is limited; ControlNet++'s cycle-consistency loss only works at low-noise timesteps, and DPO over generated image pairs is noisy and expensive to curate.
Approach
Perform preference learning over control conditions rather than generated images: the ground-truth condition c^w is preferred over a perturbed condition c^l extracted from an image generated by c^w, fixing the image so only controllability varies, which yields a lower-variance objective and cheaper dataset curation (one sample per pair).
Key results
- Achieves ~10% error rate reduction on ADE20K segmentation and 70-80% on COCO-Pose and HumanArt relative to ControlNet++ base, per the paper's oracle-relative error definition.
- On MultiGen-20M, reports 2-5% error rate reduction on Canny, Lineart, HED, and depth; depth RMSE 27.49 vs ControlNet++ 28.32.
- In a 15-annotator user study, CPO won 47.73% on HED and 46.67% on depth versus ControlNet++ at 37.87% and 40.27%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune ControlNet++ with CPO | SD1.5 controllable generation · ADE20K | mIoU: 43.64 -> 44.81 | Helped |
| fine-tune ControlNet++ with CPO | SD1.5 controllable generation · COCO-Stuff | mIoU: 34.56 -> 35.49 | Helped |
| fine-tune ControlNet with CPO | SD1.5 pose generation · COCO-Pose | mAP: 72.53 -> 87.98 | Helped |
| fine-tune ControlNet with CPO | SD1.5 pose generation · HumanArt | mAP: 34.52 -> 45.71 | Helped |
| fine-tune ControlNet++ with CPO | SD1.5 controllable generation · MultiGen-20M Canny | F1: 38.03 -> 39.28 | Helped |
| fine-tune ControlNet++ with CPO | SD1.5 controllable generation · MultiGen-20M HED | SSIM: 0.8097 -> 0.8201 | Helped |
| fine-tune ControlNet++ with CPO | SD1.5 controllable generation · MultiGen-20M Lineart | SSIM: 0.8399 -> 0.8447 | Helped |
| fine-tune ControlNet++ with CPO | SD1.5 controllable generation · MultiGen-20M Depth | RMSE: 28.32 -> 27.49 | Helped |
| add DINO-v2 feature extraction (ablation) | CPO · controllability across benchmarks | controllability | Mixed |
| increase CFG from 3.0–4.0 to 7.5 (ablation) | CPO · ADE-20K and Canny | controllability: 46.40 -> 44.81; 39.69 -> 39.28 | Hurt |
| remove the margin (ablation) | CPO · ADE-20K | mIoU: 44.81 -> 44.83 | No clear effect |
| remove pretraining regularization (λ=0) (ablation) | CPO · ADE-20K | mIoU: 44.81 -> 46.01 | Helped |
Surprising findings
- Removing the pretraining regularization (lambda=0) substantially improves controllability (~10% error reduction) but degrades FID and CLIP, and the authors argue the visual quality drop is not always perceptible.
Limitations
- Limited to fixed 512x512 resolution for ControlNet-based variants; no multi-resolution support.
- Struggles with extremely dense control conditions and fine details of small human faces.
- Gains on fine-grained edge controls (Canny, Lineart) are modest (about 2%).
Experimental setup
Datasets: ADE20K COCO-Stuff MultiGen-20M COCO-Pose HumanArt LAION-5B
Benchmarks: ADE20K COCO-Stuff COCO-Pose HumanArt MultiGen-20M
Metrics: mIoU mAP F1 SSIM RMSE FID CLIP score
Baselines: ControlNet ControlNet++ ControlAR T2I-Adapter GLIGEN Uni-ControlNet UniControl
Builds on: Diffusion-DPO ControlNet ControlNet++ DDPM
Closest papers by meaning
- Simplifying Control Mechanism in Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.87)
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.87)
- Minimal Impact ControlNet: Advancing Multi-ControlNet Integration (ICLR 2025 · similarity 0.86)
- Local Conditional Controlling for Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.86)
- Boost Your Human Image Generation Model via Direct Preference Optimization (CVPR 2025 · similarity 0.85)
- Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.