Research map / Detection and segmentation / Open-set and few-shot detection
Easing Concept Bleeding in Diffusion via Entity Localization and Anchoring
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Entity Localization and Anchoring (ELA), a training-free inference-time method that localizes entity regions from cross-attention maps of Stable Diffusion and updates latents with a concentration/attenuation loss to reduce concept bleeding.
Problem
Text-to-image diffusion models suffer concept bleeding (entity leakage, attribute binding errors) and missing entities when prompts contain multiple objects, due to cross-attention maps spilling into background and other entities.
Approach
Localize each entity's region by differencing its cross-attention map against other entities and the SoT background map, then anchor entity attention inside the mask by gradient updates to the latent during early denoising steps, with no training required.
Key results
- On full-prompt entity evaluation, ELA reaches 0.7964 CLIP text-image Full Prompt vs 0.7571 for Stable Diffusion and 0.7638 for Attend-and-Excite (Table 1).
- Animal-animal Min. Entity CLIP text-text score rises to 0.8559 (vs 0.8312 Attend-and-Excite, 0.7837 SD).
- Human evaluation: ELA gets 34.02 on ALL vs 18.05 for Attend-and-Excite and 1.80 for Stable Diffusion; on binding prompts it trails Linguistic Binding (24.20 vs 31.76).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply entity localization and anchoring to cross-attention maps | Stable Diffusion V-1.4 · Animal-animal pairs | CLIP text-text similarity: 0.7571 ± 0.0942 -> 0.7964 ± 0.0822 | Helped |
| apply entity localization and anchoring to cross-attention maps | Stable Diffusion V-1.4 · Animal-object pairs | CLIP text-text similarity: 0.7837 ± 0.1030 -> 0.8559 ± 0.0791 | Helped |
| apply entity localization and anchoring to cross-attention maps | Stable Diffusion V-1.4 · Object-object pairs | CLIP text-text similarity: 0.7350 ± 0.1003 -> 0.8135 ± 0.0919 | Helped |
| apply entity localization and anchoring | Stable Diffusion V-1.4 · All prompts | CLIP text-image similarity (Full Prompt): 0.3258 -> 0.3307 | Helped |
| apply entity localization and anchoring | Stable Diffusion V-1.4 · All prompts | CLIP text-image similarity (Min. Entity): 0.2362 -> 0.2429 | Helped |
| increase the attenuation weight λ (ablation) | ELA · Multi-entity prompts | entity leakage | Helped |
| set λ exclusively to 1, minimizing attenuation without concentration (ablation) | ELA · Multi-entity prompts | entity generation | Hurt |
| use UP-1 attention maps instead of the full method's attention maps (ablation) | ELA · Animal-animal pairs | CLIP text-text similarity: 0.7964 -> 0.7651 | Hurt |
| use UP-2 attention maps instead of the full method's attention maps (ablation) | ELA · Animal-animal pairs | CLIP text-text similarity: 0.7964 -> 0.7724 | Hurt |
| use UP-1 and UP-2 attention maps (ablation) | ELA · Animal-animal pairs | CLIP text-text similarity: 0.7964 -> 0.7930 | Hurt |
Surprising findings
- Linguistic Binding outperforms ELA on attribute binding in human evaluation (31.76 vs 24.20 on the Binding set).
- Setting lambda too high (only attenuation) causes entities to be omitted, so concentration and attenuation must be balanced.
Limitations
- Performance degrades with four or more entities or added attributes in complex scenes.
- Adds computational overhead at inference because latents are optimized during sampling.
- Attribute binding is only partially addressed and trails Linguistic Binding in human evaluation.
Experimental setup
Benchmarks: Entity-pair prompts (animal-animal, animal-object, object-object) Attribute binding prompts
Metrics: CLIP text-image similarity CLIP text-text similarity (BLIP captions) human preference assessment
Baselines: Stable Diffusion StructureDiffusion Attend-and-Excite Linguistic Binding
Builds on: Attend-and-Excite Prompt-to-Prompt Stable Diffusion
Closest papers by meaning
- Concept Replacer: Replacing Sensitive Concepts in Diffusion Models via Precision Localization (CVPR 2025 · similarity 0.88)
- Unveiling Concept Attribution in Diffusion Models (NeurIPS 2025 · similarity 0.88)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.87)
- Precise Parameter Localization for Textual Generation in Diffusion Models (ICLR 2025 · similarity 0.87)
- On Mechanistic Knowledge Localization in Text-to-Image Generative Models (ICML 2024 · similarity 0.86)
- Mechanisms of Object Localization in Vision-Language Models (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.