Research map / Image generation / Text-to-image diffusion
Text Embedding is Not All You Need: Attention Control for Text-to-Image Semantic Alignment with Text Self-Attention Maps
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes T-SAM, a training-free test-time method that optimizes diffusion latents so cross-attention token similarity matches text encoder self-attention maps, improving text-image alignment without external parsers.
Problem
Text-to-image diffusion models suffer semantic misalignment (missing objects, attribute mis-binding) because cross-attention maps of syntactically related tokens are not properly spatially aligned, and CLIP text embeddings fail to encode syntactic relations.
Approach
Text self-attention maps from the text encoder capture syntactic relations that text embeddings lose (attention sink on bos token); the method transfers these relations to cross-attention by minimizing a loss between the cross-attention cosine-similarity matrix and the text self-attention matrix, updating latent noise at inference time.
Key results
- On TIFA, T-SAM on SD v1.5 reaches 0.83 TIFA score versus 0.79 for SD and 0.80 for Linguistic Binding.
- On Objects and Animals-Objects prompts, T-SAM on SD performs comparably to CONFORM (CLIP image-text 0.36/0.37 vs 0.36/0.36) and beats LB and A&E.
- On PixArt-alpha with Attend-n-Excite prompts, T-SAM improves CLIP-T from 0.81 to 0.82 (Objects) and 0.85 to 0.86 (Animals-Objects).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| align cross-attention similarity with text self-attention maps | SD · TIFA benchmark | TIFA: 0.79 -> 0.83 | Helped |
| align cross-attention similarity with text self-attention maps | SD · TIFA benchmark | CLIP-I: 0.33 -> 0.34 | Helped |
| align cross-attention similarity with text self-attention maps | SD · TIFA benchmark | CLIP-T: 0.77 -> 0.77 | No clear effect |
| align cross-attention similarity with text self-attention maps | SD · Animals-Objects prompts | CLIP-I (image-text prompt): 0.34 -> 0.37 | Helped |
| align cross-attention similarity with text self-attention maps | SD · Objects prompts | CLIP-I (prompt-caption): 0.25 -> 0.28 | Helped |
| align cross-attention similarity with text self-attention maps | SD · Animals-Objects prompts | CLIP-I (prompt-caption): 0.26 -> 0.28 | Helped |
| align cross-attention similarity with text self-attention maps | SD · Animals-Objects prompts | CLIP-T (full prompt): 0.80 -> 0.85 | Helped |
| align cross-attention similarity with text self-attention maps | PixArt-ϖ · Objects prompts | CLIP-T: 0.81 -> 0.82 | Helped |
| align cross-attention similarity with text self-attention maps | PixArt-ϖ · Animals-Objects prompts | CLIP-T: 0.85 -> 0.86 | Helped |
| align cross-attention similarity with text self-attention maps | SD · Objects prompts | CLIP-T (full prompt): 0.76 -> 0.81 | Helped |
Surprising findings
- Text embedding similarity does not correlate well with syntactic bindings, while text self-attention maps do, attributed to the attention sink on the bos token.
- Linguistic Binding produces images identical to SD when the text parser extracts no relations.
Limitations
- Inaccuracies in object counting
- Potential noise introduced by text attention maps
- Requires per-step latent optimization at inference, adding cost
Experimental setup
Datasets: Attend-n-Excite prompt sets TIFA v1.0 prompts
Benchmarks: TIFA v1.0 Attend-n-Excite prompts
Metrics: TIFA score CLIP-I CLIP-T user preference
Baselines: Stable Diffusion Linguistic Binding Attend-n-Excite CONFORM
Builds on: Prompt-to-Prompt Attend-and-Excite CONFORM Linguistic Binding
Closest papers by meaning
- Token-to-Token Alignment of Text Embeddings for Semantic Blending (NeurIPS 2026 · similarity 0.90)
- CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching (NeurIPS 2024 · similarity 0.89)
- Aligning Text to Image in Diffusion Models is Easier Than You Think (NeurIPS 2025 · similarity 0.89)
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models (AAAI 2024 · similarity 0.88)
- AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.