Research map / Image generation / Text-to-image diffusion
CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free test-time method that applies an InfoNCE contrastive loss on cross-attention maps of object and attribute tokens (across adjacent timesteps) to improve prompt fidelity in Stable Diffusion and Imagen.
Problem
Text-to-image diffusion models often miss objects, mis-bind attributes to subjects, and miscount objects relative to the prompt, and prior fixes rely on task-specific tailored objectives.
Approach
Treat each object and its attribute as positive pairs and other object-attribute combinations as negatives, then apply an InfoNCE loss to 16x16 cross-attention maps at timesteps t and t+1 to separate objects and keep attributes bound to them, updating latents at test time.
Key results
- On Stable Diffusion, TIFA Obj-Obj rises to 0.88 for CONFORM vs 0.65 for SD and 0.83 for Divide & Bind.
- In a 25-person user study on SD, CONFORM received 72% to 94% of votes across benchmark sets.
- On Imagen, CONFORM reaches TIFA 0.91 on Obj-Obj versus 0.88 for Imagen and Attend & Excite, and was preferred 96% to 98% in the user study.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CONFORM instead of D&B | Stable Diffusion · Animal-Animal | CLIP text-text similarity: 0.81 -> 0.82 | Helped |
| Apply CONFORM instead of A&E | Stable Diffusion · Animal-Object | CLIP text-text similarity: 0.83 -> 0.85 | Helped |
| Apply CONFORM instead of A&E | Stable Diffusion · Object-Object | CLIP text-text similarity: 0.81 -> 0.82 | Helped |
| Apply CONFORM instead of D&B | Stable Diffusion · Animal-Animal | TIFA: 0.93 -> 0.95 | Helped |
| Apply CONFORM instead of D&B | Stable Diffusion · Animal-Object | TIFA: 0.91 -> 0.94 | Helped |
| Apply CONFORM instead of D&B | Stable Diffusion · Object-Object | TIFA: 0.83 -> 0.88 | Helped |
| Apply CONFORM instead of D&B | Stable Diffusion · Multi-Obj | TIFA: 0.73 -> 0.74 | Helped |
| Apply CONFORM instead of Attend & Excite | Imagen · Animal-Animal | TIFA: 0.84 -> 0.84 | No clear effect |
| Apply CONFORM instead of Imagen | Imagen · Animal-Object | TIFA: 0.93 -> 0.94 | Helped |
| Apply CONFORM instead of Imagen | Imagen · Object-Object | TIFA: 0.88 -> 0.91 | Helped |
| Apply CONFORM instead of Imagen | Imagen · Multi-Obj | TIFA: 0.73 -> 0.76 | Helped |
Surprising findings
- Authors note CLIP-based metrics should be interpreted cautiously because the compared models are already conditioned on CLIP embeddings.
Limitations
- Struggles when initial attention maps exclude key objects (e.g., missing bear in SD).
- In Imagen, refinement can sometimes separate objects unnaturally.
- Relies on CLIP-based metrics that are biased because compared models use CLIP embeddings.
Experimental setup
Datasets: Attend-and-Excite prompt protocols Divide-and-Bind multi-instance prompts
Benchmarks: TIFA CLIP similarity (full and min object) BLIP caption text-text similarity user study
Metrics: TIFA score CLIP-full similarity CLIP-min similarity text-text similarity user preference rate
Baselines: Stable Diffusion Attend & Excite Divide & Bind ComposableDiffusion StructureDiffusion Imagen
Builds on: Attend-and-Excite Divide-and-Bind InfoNCE Hertz prompt-to-prompt attention analysis
Closest papers by meaning
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.91)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.91)
- Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function (NeurIPS 2024 · similarity 0.91)
- Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.90)
- Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers (CVPR 2024 · similarity 0.90)
- Diffusion PID: Interpreting Diffusion via Partial Information Decomposition (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.