Research map / Image generation / Text-to-image diffusion
Cross-Attention Head Position Patterns Can Align with Human Visual Concepts in Text-to-Image Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Constructs head relevance vectors (HRVs) mapping cross-attention heads of Stable Diffusion to 34 human-specified visual concepts, validates them via ordered weakening, and uses concept strengthening/adjusting to improve polysemy handling, image editing (P2P-HRV) and multi-concept generation (A&E-HRV).
Problem
Cross-attention layers in text-to-image diffusion models are poorly understood at the head level, and existing methods steer whole CA layers without fine-grained per-head control over human-specified visual concepts.
Approach
Builds a per-concept vector over all cross-attention heads by counting, for each random generation and timestep, which concept's token attention is strongest at each head; validates it with ordered weakening of heads and reuses the vectors as per-head rescaling factors for concept strengthening and adjusting.
Key results
- Polysemous word misinterpretation rate dropped from 63.0% (SD) to 15.9% (SD-HRV) in human evaluation.
- P2P-HRV improved CLIP score by 4.20% on Image Style and 9.91% on Weather Conditions over PnP, with about 2.4x and 2.5x human preference votes versus second-best.
- A&E-HRV improved over Attend-and-Excite by 4.5% to 6.3% on Type 1 and 2.3% to 5.1% on Type 2 prompts across metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply concept adjusting to polysemous-word prompts | Stable Diffusion v1.4 · Polysemous-word prompts | Human perceived misinterpretation rate: 63.0\% to 15.9\% | Helped |
| Add concept strengthening to Attend-and-Excite | Multi-concept generation Type 1 · Multi-concept generation Type 1 | Full Prompt: 0.3544 -> 0.3702~(+4.5\%) | Helped |
| Add concept strengthening to Attend-and-Excite | Multi-concept generation Type 1 · Multi-concept generation Type 1 | Min. Object: 0.2017 -> 0.2078~(+3.0\%) | Helped |
| Add concept strengthening to Attend-and-Excite | Multi-concept generation Type 1 · Multi-concept generation Type 1 | BLIP-score: 0.7049 -> 0.7491~(+6.3\%) | Helped |
| Add concept strengthening to Attend-and-Excite | Multi-concept generation Type 2 · Multi-concept generation Type 2 | Full Prompt: 0.3883 -> 0.3971~(+2.3\%) | Helped |
| Add concept strengthening to Attend-and-Excite | Multi-concept generation Type 2 · Multi-concept generation Type 2 | Min. Object: 0.1972 -> 0.2073~(+5.1\%) | Helped |
| Add concept strengthening to Attend-and-Excite | Multi-concept generation Type 2 · Multi-concept generation Type 2 | BLIP-score: 0.6373 -> 0.6580~(+3.2\%) | Helped |
| Add concept strengthening to P2P | Image editing · Image Style | CLIP: 0.3286 -> 0.3424 | Helped |
| Add concept strengthening to P2P | Image editing · Weather Conditions | CLIP: 0.3046 -> 0.3348 | Helped |
| Use concept adjusting instead of concept strengthening (ablation) | SD-HRV · Polysemous-word prompts | Misinterpretation | Helped |
| Order heads by HRV relevance rather than random order (ablation) | Ordered weakening analysis · Six visual concepts | LeRHF − MoRHF area: 10.12 vs 0.62 | Helped |
Surprising findings
- Argmax-before-summation was needed because raw CA map scales differ across heads by up to 8.1x, so naive summation would be dominated by large-scale heads.
Limitations
- Concept-adjusting for polysemy requires manual specification of target token and desired/undesired concepts.
- Some failures stem from the underlying T2I model and others from the concept-words or algorithm; e.g., 'rusted nut' only partially resolved.
- LeRHF weakening interpretation is confounded by least relevant heads affecting other concepts.
Experimental setup
Datasets: ImageNet class names PromptHero prompts
Benchmarks: P2P editing prompts (own set) Attend-and-Excite multi-concept prompts (Type 1 and Type 2) polysemous word prompts (own set of 10)
Metrics: CLIP image-text similarity BG-DINO score human preference score full prompt similarity minimum object similarity BLIP-score human perceived misinterpretation rate CLIP similarity change under ordered weakening
Baselines: Stable Diffusion Structured Diffusion Attend-and-Excite SDEdit Prompt-to-Prompt PnP MasaCtrl FPE
Builds on: Prompt-to-Prompt Attend-and-Excite Stable Diffusion P2P rescaling
Closest papers by meaning
- Head Pursuit: Probing Attention Specialization in Multimodal Transformers (NeurIPS 2025 · similarity 0.89)
- Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.87)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.87)
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.86)
- Localizing and Editing Knowledge In Text-to-Image Generative Models (ICLR 2024 · similarity 0.86)
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.