Research map / Image generation / Text-to-image diffusion
Progressive Compositionality in Text-to-Image Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds ConPair, a 15k-pair dataset of minimally different contrastive images generated with LLM prompts and VQA checking, and fine-tunes diffusion models with a three-stage curriculum and an InfoNCE-style contrastive loss (EvoGen).
Problem
Text-to-image diffusion models often fail at compositional generation, including attribute binding, counting, and object relationships, especially for complex prompts; existing methods assume fixed compositional structures and lack curated contrastive image data.
Approach
Generate a curated contrastive image dataset (ConPair) with minimal visual differences using LLM-composed prompts and VQA-guided selection, then fine-tune diffusion models with a contrastive loss over positive/negative image pairs in a three-stage curriculum from simple single-object to complex multi-object scenes.
Key results
- On T2I-CompBench, EvoGen reaches 71.04 color, 72.34 texture, 42.52 complex, versus 50.65/49.22/33.86 for SD v2 and 68.86/70.44/41.17 for PixArt-alpha.
- On Gen-AI, EvoGen-SD3-Medium averages 0.72 on Advanced prompts, above DALL-E 3's 0.65, while trailing DALL-E 3 on some basic categories.
- Ablation shows ConPair fine-tuning alone raises SD v2 color from 50.65 to 63.63; adding contrastive loss gives 69.45 and multi-stage fine-tuning gives 71.04.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune Stable Diffusion v2 directly on ConPair (ablation) | Stable Diffusion v2 · T2I-CompBench Color | alignment score: 50.65 -> 63.63 | Helped |
| add contrastive loss to ConPair fine-tuning (ablation) | Stable Diffusion v2 fine-tuned on ConPair · T2I-CompBench Color | alignment score: 63.63 -> 69.45 | Helped |
| add multi-stage fine-tuning after ConPair and contrastive loss (ablation) | EvoGen · T2I-CompBench Complex | alignment score: 38.14 -> 42.52 | Helped |
| add multi-stage fine-tuning after ConPair and contrastive loss (ablation) | EvoGen · T2I-CompBench Non-Spatial | alignment score: 32.09 -> 33.08 | Helped |
| use EvoGen instead of DALL-E 3 | Text-to-image generation · Gen-AI Basic | average score: 0.90 -> 0.89 | Hurt |
| use EvoGen instead of SD3-Medium | Text-to-image generation · Gen-AI Advanced | average score: 0.68 -> 0.72 | Helped |
| fine-tune with ConPair rather than COLA | EvoGen · T2I-CompBench Complex | alignment score: 33.15 -> 42.52 | Helped |
| fine-tune with ConPair rather than BISON | EvoGen · T2I-CompBench Complex | alignment score: 32.91 -> 42.52 | Helped |
| use EvoGen rather than CONFORM | Stable Diffusion-based generation · Attn-Exct benchmark Animal-Animal | CLIP similarity: 0.82 -> 0.84 | Helped |
| use EvoGen rather than CONFORM | Stable Diffusion-based generation · Attn-Exct benchmark Animal-Obj | CLIP similarity: 0.85 -> 0.86 | Helped |
| use EvoGen rather than CONFORM | Stable Diffusion-based generation · Attn-Exct benchmark Obj-Obj | CLIP similarity: 0.82 -> 0.85 | Helped |
Surprising findings
- Contrastive loss helps attribute binding strongly but has less impact on spatial and complex categories, which the authors attribute to relationship differences being harder to detect.
- Shape performance of EvoGen (54.57) is slightly below PixArt-alpha (55.82) and MARS (54.31) on T2I-CompBench.
Limitations
- Dataset coverage of compositional scenarios could be expanded.
- VQA-guided generation does not fully guarantee image faithfulness, especially for complex prompts.
- Aesthetic quality may be slightly lower than other models per the user study.
Experimental setup
Datasets: ConPair T2I-CompBench Gen-AI COLA BISON
Benchmarks: T2I-CompBench Gen-AI Attn-Exct benchmark ConPair alignment
Metrics: T2I-CompBench attribute/relation/complex scores CLIP similarity Gen-AI accuracy user preference rate
Baselines: Stable Diffusion v1.4 Stable Diffusion v2 DALL-E 2 SDXL ComposableDiffusion v2 StructureDiffusion v2 Attn-Exct v2 GORs PixArt-alpha MARS COLA BISON
Builds on: Stable Diffusion InfoNCE MagicBrush T2I-CompBench
Closest papers by meaning
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.89)
- ORCA: Hunting Compositional Failures in Text-to-Image Diffusion (NeurIPS 2026 · similarity 0.89)
- Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding (CVPR 2024 · similarity 0.89)
- ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty (NeurIPS 2024 · similarity 0.89)
- Localizing and Editing Knowledge In Text-to-Image Generative Models (ICLR 2024 · similarity 0.89)
- IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.