Research map / Image generation / Text-to-image diffusion
CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Stable Diffusion (SDXL and SD1.5) with LoRA using an image-to-text captioning likelihood loss, an adversarial fidelity term, and attention-to-segmentation-mask losses to improve text-image alignment.
Problem
Text-to-image diffusion models often ignore concepts in prompts (concept ignorance) or bind them to the wrong image regions (concept mismapping), causing text-image misalignment.
Approach
Treat the prompt as a supervisory signal: score generated images with a frozen image-to-text captioner's log-likelihood of the prompt (concept activation), and supervise cross-attention maps of entity and attribute tokens against segmentation masks of generated images (attribute concentration), while a discriminator built from the pre-trained UNet and mixed noisy-real latents preserve fidelity.
Key results
- CoMat-SDXL raises T2I-CompBench color binding from 0.5879 (SDXL) to 0.7827 and TIFA from 85.9 to 87.5.
- CoMat-SD1.5 improves DPG-Bench from 63.18 to 73.32 and TIFA from 78.4 to 85.8.
- Ablation shows concept activation gives most of the gain; fidelity preservation with mixed latent lowers FID-10K from 19.02 to 15.43 relative to the unpreserved variant (baseline SD1.5 16.69).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune SDXL with CoMat | SDXL · T2I-CompBench Color | score: 0.5879 -> 0.7827 | Helped |
| fine-tune SDXL with CoMat | SDXL · TIFA | score: +1.6 | Helped |
| fine-tune SDXL with CoMat | SDXL · DPG-Bench | score: 74.65 -> 77.13 | Helped |
| fine-tune SDXL with CoMat | SDXL · T2I-CompBench Non-Spatial | score: 0.3119 -> 0.3187 | Mixed |
| add concept activation (ablation) | SDXL · T2I-CompBench Color | score: 0.5879 -> 0.7455 | Helped |
| add attribute concentration on top of concept activation (ablation) | SDXL with concept activation · T2I-CompBench Color | score: 0.7455 -> 0.7827 | Helped |
| remove fidelity preservation and mixed latent (ablation) | CoMat-SD1.5 · COCO validation FID-10K | FID: 16.69 -> 19.02 | Hurt |
| use real-world latent instead of generated latent for discriminator input (ablation) | CoMat-SD1.5 · COCO validation FID-10K | FID: 16.69 -> 17.99 | Hurt |
| replace UNet discriminator with DINO (ablation) | CoMat-SD1.5 · COCO validation FID-10K | FID: 16.69 -> 23.86 | Hurt |
| add mixed latent training (ablation) | CoMat-SD1.5 with UNet discriminator · COCO validation FID-10K | FID: 16.69 -> 15.43 | Helped |
| use BLIP as the image-to-text model rather than GIT or LLaVA (ablation) | CoMat-SDXL · T2I-CompBench Color | score: 0.6916 -> 0.7827 | Helped |
| use GIT as the image-to-text model (ablation) | CoMat-SDXL · T2I-CompBench Color | score: 0.5879 -> 0.6916 | Helped |
Surprising findings
- Without fidelity preservation, the image-to-text reward alone degrades FID from 16.69 to 19.02, showing reward hacking.
- Image-to-text model choice matters: BLIP outperforms GIT and the 7.2B LLaVA on T2I-CompBench despite being smaller, and sensitivity scores track the gains.
Limitations
- Attribute concentration cannot assign attributes when multiple same-name objects appear, since segmentation cannot separate them.
- Training is costly because the full denoising process must be run with gradients, extending training time.
- Uses a BLIP captioner rather than MLLMs, which the authors note may give finer-grained guidance.
Experimental setup
Datasets: T2I-CompBench training set HRS-Bench ABC-6K COCO
Benchmarks: T2I-CompBench TIFA DPG-Bench COCO FID-10K ARO
Metrics: T2I-CompBench color/shape/texture/spatial/non-spatial/comple TIFA score DPG score FID-10K attack success rate user preference
Baselines: StructureDiffusion Composable Diffusion Attend-and-Excite TokenCompose PixArt-alpha Playground-v2 SD1.5 SDXL
Builds on: Stable Diffusion SDXL Attend-and-Excite TokenCompose BLIP
Closest papers by meaning
- Information Theoretic Text-to-Image Alignment (ICLR 2025 · similarity 0.90)
- Aligning Text to Image in Diffusion Models is Easier Than You Think (NeurIPS 2025 · similarity 0.89)
- Text Embedding is Not All You Need: Attention Control for Text-to-Image Semantic Alignment with Text Self-Attention Maps (CVPR 2025 · similarity 0.89)
- LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation (AAAI 2025 · similarity 0.89)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.89)
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.