Research map / Image generation / Text-to-image diffusion
Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BoxNet, a per-timestep box predictor trained on frozen Stable Diffusion features, plus unique-mask cross- and self-attention control, as a training-light plugin that reduces attribute leakage, entity leakage, and missing entities in T2I generation.
Problem
Text-to-image diffusion models such as Stable Diffusion misbind attributes to entities, leak entities into one another, and omit entities mentioned in compositional prompts.
Approach
A BoxNet transformer, attached to frozen SD U-Net features at each denoising step, predicts a box per parsed entity; these boxes become unique Gaussian-argmax masks that constrain cross-attention token maps and self-attention maps during sampling, without fine-tuning SD.
Key results
- The full method reaches a Min. Object Score of 0.6028 on COCO and 0.5991 on NON-COCO, versus 0.3973 and 0.3998 for Stable Diffusion.
- Subjective Fidelity Score on COCO improves to 0.4331 from 0.3021 for Stable Diffusion, with FID 17.47 versus 17.79.
- Plugging the method into GLIGEN raises COCO Min. Object Score from 0.5046 (BASE) to 0.6718 (with cross- and self-attention control).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add BoxNet and non-unique cross-attention mask control (ablation) | Stable Diffusion · COCO | Min. Object Score: 0.3973 ± 0.0021 -> 0.4010 ± 0.0028 | No clear effect |
| Add unique mask control (ablation) | Stable Diffusion with BoxNet and cross-attention control · COCO | Min. Object Score: 0.4010 ± 0.0028 -> 0.4456 ± 0.0039 | Helped |
| Add unique mask control (ablation) | Stable Diffusion with BoxNet and cross-attention control · NON-COCO | Min. Object Score: 0.4307 ± 0.0042 -> 0.4779 ± 0.0055 | Helped |
| Add self-attention control (ablation) | Stable Diffusion with BoxNet and unique cross-attention masks · COCO | Min. Object Score: 0.4456 ± 0.0039 -> 0.6028 ± 0.0047 | Helped |
| Add self-attention control (ablation) | Stable Diffusion with BoxNet and unique cross-attention masks · NON-COCO | Min. Object Score: 0.4779 ± 0.0055 -> 0.5991 ± 0.0044 | Helped |
| Add cross- and self-attention mask control | Stable Diffusion · COCO | Min. Object Score: 0.3973 ± 0.0021 -> 0.6028 ± 0.0047 | Helped |
| Add cross- and self-attention mask control | Stable Diffusion · NON-COCO | Min. Object Score: 0.3998 ± 0.0048 -> 0.5991 ± 0.0044 | Helped |
| Add cross- and self-attention mask control | AAE · NON-COCO | Min. Object Score: 0.4338 ± 0.0021 -> 0.5918 ± 0.0028 | Mixed |
| Add cross- and self-attention mask control | GLIGEN · COCO | Min. Object Score: 0.5046 ± 0.0022 -> 0.6718 ± 0.0045 | Helped |
| Add cross- and self-attention mask control | GLIGEN · NON-COCO | Min. Object Score: 0.4574 ± 0.0059 -> 0.6839 ± 0.0024 | Helped |
| Add cross- and self-attention mask control | Stable Diffusion · COCO | FID: 17.79 -> 17.47 | Helped |
Surprising findings
- Applying the method to AAE on NON-COCO unexpectedly lowers performance relative to AAE alone, attributed to conflict between BoxNet boxes and AAE's regions of interest.
- CLIP was found to discriminate colors poorly, with only about 43% correctness in a check on 200 entities, so CLIP scores were not used as the main metric.
Limitations
- Self-attention control can introduce image-quality degradation such as unnatural entity-background integration or background tearing.
- Gains on NON-COCO shrink or reverse when combined with AAE, due to box–region conflicts.
- Grounding DINO and CLIP are imperfect for attribute evaluation, so attribute correctness relies on a small user study of 11 respondents.
Experimental setup
Datasets: COCO 2014
Benchmarks: Self-built color-entity benchmark (COCO and NON-COCO categories) FID on 10k COCO validation prompts
Metrics: Min. Object Score (Grounding DINO) Subjective Fidelity Score (user study) Avg CLIP image-text Avg CLIP text-text FID
Baselines: Stable Diffusion StructureDiffusion Attend-and-Excite
Builds on: Stable Diffusion Attend-and-Excite DETR GLIGEN
Closest papers by meaning
- Grounded Text-to-Image Synthesis with Attention Refocusing (CVPR 2024 · similarity 0.91)
- Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions (CVPR 2025 · similarity 0.89)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.89)
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.