Research map / Image generation / Text-to-image diffusion
Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Direct Consistency Optimization (DCO), a DPO-style fine-tuning loss that penalizes deviation from the pretrained T2I model on reference images, enabling more robust subject/style personalization and direct merging of subject and style LoRAs.
Problem
Low-shot fine-tuning of text-to-image diffusion models for personalization overfits and forgets pretrained knowledge, reducing image-text alignment and compositional ability; prior preservation loss needs extra prior data and hurts subject fidelity.
Approach
Derives a log-sigmoid loss from a KL-constrained reward maximization view, where the implicit reward is consistency with reference images and the deviation from the pretrained model is measured via noise-prediction differences; it adaptively down-weights updates when the fine-tuned model deviates too far from the pretrained one. Also introduces consistency guidance sampling and comprehensive capti
Key results
- On my-subject-in-my-style generation, DCO Merge reaches subject similarity 0.462 and text similarity 0.773, versus 0.386/0.430 for DB Merge and 0.406/0.729 for DB ZipLoRA.
- In user study, DCO wins 72.7% of prompt-fidelity judgments against DB and 81.9% of subject-fidelity judgments against DB+p.p.
- Compared with DB, DCO positions on a better Pareto frontier of image-text and image similarity across subject customization and stylization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune subject and style LoRAs with DCO instead of DreamBooth, then arithmetic merge | Subject and style LoRAs · My subject in my style | Subject similarity: 0.386 -> 0.462; 0.406 -> 0.462 | Helped |
| Fine-tune subject and style LoRAs with DCO instead of DreamBooth, then arithmetic merge | Subject and style LoRAs · My subject in my style | Image-text similarity: 0.430 -> 0.773; 0.729 -> 0.773 | Helped |
| Fine-tune subject and style LoRAs with DCO instead of DreamBooth, then arithmetic merge | Subject and style LoRAs · My subject in my style | Style similarity: 0.672 -> 0.651; 0.662 -> 0.651 | Hurt |
| Use DCO instead of DreamBooth | DreamBooth subject personalization · Subject customization and stylization | Image-text similarity and image similarity | Helped |
| Use DCO instead of DreamBooth with prior preservation | Subject personalization · Subject customization and stylization | Image similarity | Helped |
| Early stop DreamBooth at half the training steps (ablation) | DreamBooth with comprehensive captions · Subject customization | Image-text similarity: 0.754 -> 0.842 | Helped |
| Early stop DreamBooth at half the training steps (ablation) | DreamBooth with comprehensive captions · Subject customization | Image similarity: 0.754 -> 0.575 | Hurt |
| Early stop DreamBooth at half the training steps (ablation) | DreamBooth with comprehensive captions · Subject stylization | Image-text similarity: 0.468 -> 0.645 | Helped |
| Early stop DreamBooth at half the training steps (ablation) | DreamBooth with comprehensive captions · Subject stylization | Image similarity: 0.638 -> 0.630 | Hurt |
| Early stop DreamBooth at half the training steps (ablation) | DreamBooth with comprehensive captions · Subject customization and stylization | Pareto frontier | No clear effect |
| Reduce prior preservation weight to 0.5 (ablation) | DreamBooth with prior preservation · Subject customization | Image similarity: 0.532 -> 0.547 | Helped |
| Reduce prior preservation weight to 0.5 (ablation) | DreamBooth with prior preservation · Subject customization | Image-text similarity: 0.864 -> 0.850 | Hurt |
Surprising findings
- Prior preservation loss shifts the operating point toward higher prompt fidelity at the cost of subject consistency rather than pushing the Pareto frontier outward.
- DCO fine-tuned subject and style LoRAs show no clear difference in cosine similarity versus DB LoRAs, yet merge without interference, challenging the view that LoRA cosine similarity measures interference.
Limitations
- Multi-subject composition is unstable, with learned concepts often mixing, especially for similar subjects.
- Extra computation: about 1.3x fine-tuning time and 2x sampling time for consistency guidance.
- Evaluation metrics are imperfect, so results rely on Pareto curves over guidance scales rather than single operating points.
Experimental setup
Datasets: DreamBooth dataset StyleDrop style images Parti prompts
Benchmarks: DreamBooth dataset StyleDrop
Metrics: DINOv2 image similarity SigLIP image-text similarity CLIP-T DINO user preference win rate
Baselines: DreamBooth (DB) DreamBooth with prior preservation loss (DB+p.p.) ZipLoRA DB early stopping (DB+ES) DB Merge
Builds on: DreamBooth Direct Preference Optimization LoRA Textual Inversion ZipLoRA
Closest papers by meaning
- Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation (ICLR 2024 · similarity 0.91)
- PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction (ICLR 2025 · similarity 0.90)
- Aligning Text to Image in Diffusion Models is Easier Than You Think (NeurIPS 2025 · similarity 0.90)
- Improving Long-Text Alignment for Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.90)
- Orthogonal Adaptation for Modular Customization of Diffusion Models (CVPR 2024 · similarity 0.89)
- Energy-Guided Optimization for Personalized Image Editing with Pretrained Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.