Research map / Image generation / Text-to-image diffusion
Grounded Text-to-Image Synthesis with Attention Refocusing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free attention-refocusing losses (cross- and self-attention) applied during diffusion sampling, paired with GPT-4 layout generation, to improve layout-grounded text-to-image synthesis.
Problem
Text-to-image diffusion models fail to faithfully render prompts with multiple objects, attributes, and spatial relations, causing missing, mixed, or swapped objects.
Approach
Two losses (CAR for cross-attention and SAR for self-attention) update the noised latent at early denoising steps so that each grounded token's attention concentrates inside its box and pixels inside a box attend less to outside regions; GPT-4 supplies the boxes via in-context learning.
Key results
- On HRS, adding CAR and SAR to GLIGEN raises spatial accuracy from 30.74 to 40.22 and color accuracy from 18.78 to 26.32.
- On TIFA, attention refocusing improves Stable Diffusion 1.4 average score from 78.38 to 78.87 and counting from 68.15 to 69.37.
- On mask-guided generation, the method reaches IoU 38.97 vs 34.99 for DenseDiffusion, and FID on COCO 2014 stays near baseline (20.63 vs 20.82 for SD).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CAR and SAR losses | GLIGEN · HRS | Spatial accuracy: 30.74 -> 40.22 | Helped |
| Add CAR and SAR losses | GLIGEN · HRS | Color accuracy: 18.78 -> 26.32 | Helped |
| Add CAR and SAR losses | Stable Diffusion 1.4 · TIFA | Average score: 78.38 -> 78.87 | Helped |
| Add CAR and SAR losses | Stable Diffusion 2.1 · TIFA | Average score: 81.84 -> 81.89 | Helped |
| Apply attention-refocusing losses with mask guidance | Stable Diffusion · Mask-guidance dataset | IoU: 34.99 ± 1.13 -> 38.97 ± 0.56 | Helped |
| Add CAR and SAR losses | Stable Diffusion · COCO 2014 | FID: 20.82 -> 21.03 | Hurt |
| Add CAR and SAR losses | GLIGEN · COCO 2014 | FID: 20.63 -> 20.37 | Helped |
| Use SAR without CAR (ablation) | GLIGEN · HRS | Spatial accuracy: 30.74 -> 36.43 | Helped |
| Use CAR without SAR (ablation) | GLIGEN · HRS | Spatial accuracy: 30.74 -> 36.92 | Helped |
| Use both CAR and SAR (ablation) | GLIGEN · HRS | Spatial accuracy: 30.74 -> 40.22 | Helped |
Surprising findings
- Attention refocusing gives a larger gain on spatial accuracy for GLIGEN than CAR or SAR alone, with the combination giving about a 10-point improvement.
- Llama 2 (13B) and GPT-3 were evaluated for layout generation; GPT-4 scored highest but correctness was only 88.5%.
Limitations
- GPT-4 sometimes miscounts objects or produces small boxes for prompts with many objects.
- Grounded models may fail on out-of-distribution layouts even when GPT-4 layouts are correct.
- Per-step gradient updates add sampling cost, and code and data release is only promised for the future.
Experimental setup
Datasets: COCO 2014 DenseDiffusion mask-and-text dataset
Benchmarks: HRS DrawBench TIFA COCO 2014
Metrics: F1 accuracy TIFA score FID IoU CLIP score SOA-I Format Validness Correctness
Baselines: Stable Diffusion Attend-and-excite Layout-guidance MultiDiffusion GLIGEN BoxDiff SD-Pww DenseDiffusion ControlNet
Builds on: Attend-and-Excite Layout-guidance GLIGEN Stable Diffusion MultiDiffusion
Closest papers by meaning
- Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models (AAAI 2024 · similarity 0.91)
- Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.90)
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- Spatial Transport Optimization by Repositioning Attention Map for Training-Free Text-to-Image Synthesis (CVPR 2025 · similarity 0.89)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.89)
- Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjects (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.