Research map / Image generation / Text-to-image diffusion
Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes training-free Self-Cross diffusion guidance that penalizes overlap between cross-attention maps and aggregated self-attention maps to reduce subject mixing for similar-looking subjects in text-to-image diffusion, and releases the SSD benchmark evaluated with GPT-4o.
Problem
Text-to-image diffusion models suffer from subject mixing, especially when synthesizing multiple similar-looking subjects in one image.
Approach
Penalize overlap between each subject's cross-attention map and the aggregated self-attention map of other subjects, where the aggregation is a cross-attention-weighted sum over all Otsu-masked patches of a subject rather than only the most discriminant patch.
Key results
- On SSD-2 with SD1.4, Self-Cross reaches 86.45% w/o-mixing TIFA-GPT4o score, versus 79.70% for INITNO and 79.50% for CONFORM as reported in Table 4.
- On Animal-Animal with SD1.4, the w/o-mixing score is 92.94%, about 8.7 points above CONFORM's 84.22%.
- On SD3-medium with three-subject SSD-3 prompts, the Ext score rises from 33.54 to 57.92 and w/o-mixing from 70.08 to 77.15 (Table 1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Self-Cross guidance instead of CONFORM | Stable Diffusion 1.4 · Animal-Animal | absence of mixing: 84.22 -> 92.94 | Helped |
| Use Self-Cross guidance instead of INITNO | Stable Diffusion 1.4 · SSD-2 | absence of mixing: 79.70 -> 86.45 | Helped |
| Use Self-Cross guidance instead of SD3 Medium | Stable Diffusion 3 Medium · SSD-3 | absence of mixing: 70.08 -> 77.15 | Helped |
| Add Self-Cross guidance to CONFORM | Stable Diffusion 2.1 · SSD-2 | absence of mixing: 89.76 -> 93.20 | Helped |
| Add Self-Cross guidance to CONFORM | Stable Diffusion 2.1 · Animal-Animal | absence of mixing: 92.49 -> 94.66 | Helped |
| Add Self-Cross guidance to CONFORM | Stable Diffusion 2.1 · SSD-2 | recognizability: 53.90 -> 55.98 | Helped |
| Add Self-Cross guidance to CONFORM | Stable Diffusion 2.1 · Animal-Animal | recognizability: 70.19 -> 69.00 | Hurt |
| Use masked patches for self-attention aggregation instead of only the maximum-attention patch (ablation) | Self-Cross guidance · similar-subject image synthesis | subject mixing | Helped |
| Use masked patches for self-attention aggregation instead of the top 16 patches (ablation) | Self-Cross guidance · similar-subject image synthesis | subject mixing | Helped |
Surprising findings
- Attend-and-Excite-style existence guidance is reported to be inefficient on Stable Diffusion 2.1, so the authors apply CONFORM for early steps and Self-Cross for later steps.
Limitations
- Relies on attention-based guidance that can be inefficient for some backbones such as SD2.1, requiring a combination with CONFORM.
- Evaluation depends on GPT-4o as judge and on TIFA-style VQA; CLIP score is explicitly deemed unreliable.
- Only pairs of subjects extended to multiple subjects by averaging; video and attribute binding left for future work.
Experimental setup
Datasets: Animal-Animal Animal-Object Object-Object T2ICompBench 2D-Spatial T2ICompBench 3D-Spatial
Benchmarks: Animal-Animal Animal-Object Object-Object SSD-2 SSD-3 T2ICompBench 2D-Spatial T2ICompBench 3D-Spatial
Metrics: TIFA-GPT4o Ext TIFA-GPT4o Rec TIFA-GPT4o w/o M TIFA-GPT4o Rel text-text similarity (BLIP)
Baselines: Stable Diffusion 1.4 INITNO CONFORM Attend-and-Excite Separate-and-Enhance
Builds on: Attend-and-Excite INITNO CONFORM Stable Diffusion
Closest papers by meaning
- Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.91)
- Grounded Text-to-Image Synthesis with Attention Refocusing (CVPR 2024 · similarity 0.89)
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- OneActor: Consistent Subject Generation via Cluster-Conditioned Guidance (NeurIPS 2024 · similarity 0.89)
- $\text{I}^2\text{AM}$: Interpreting Image-to-Image Latent Diffusion Models via Bi-Attribution Maps (ICLR 2025 · similarity 0.88)
- Zigzag Diffusion Sampling: Diffusion Models Can Self-Improve via Self-Reflection (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.