Research map / Image generation / Text-to-image diffusion
Spatial Transport Optimization by Repositioning Attention Map for Training-Free Text-to-Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STORM, a training-free method that repositions cross-attention maps via optimal transport (Sinkhorn-based Spatial Transport Optimization with an ST Cost) during early denoising steps to fix spatial misplacement in Stable Diffusion.
Problem
Text-to-image diffusion models often place objects at positions that contradict spatial cues in the prompt (mislocated objects), alongside missing objects and mismatched attributes, and existing fixes rely on fixed layouts or templates.
Approach
Treats each token's cross-attention map as a distribution and uses entropic optimal transport (Sinkhorn) with a cost combining directional positional terms relative to a reference object's centroid and an overlap penalty, applied via latent gradient updates at early denoising timesteps, requiring no extra spatial inputs or weight updates.
Key results
- On VISOR with SD 1.4, STORM reaches 57.58% conditional spatial accuracy and 61.01% object accuracy versus 18.81% and 29.86% for vanilla SD 1.4.
- On T2I-CompBench with SD 2.1, STORM improves spatial score to 0.1981 from 0.1342 for SD 2.1 and reaches 0.7884 texture binding.
- In a 30-participant user study, STORM achieves 35.92% spatial correctness versus 13.20% for the best baseline (Attend-and-Excite).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply STO at timesteps 1–24 rather than no STO (ablation) | STORM · VISOR (SD 1.4) | Object Accuracy (%): 29.86 -> 61.01 | Helped |
| Apply STO earlier, at timesteps 1–24 rather than 19–24 (ablation) | STORM · VISOR (SD 1.4) | Object Accuracy (%): 46.62 -> 61.01 | Helped |
| Use Manhattan movement distance (ablation) | STORM cost function · VISOR (SD 1.4) | Object Accuracy (%): 61.01 -> 27.32 | Hurt |
| Use Euclidean movement distance (ablation) | STORM cost function · VISOR (SD 1.4) | Object Accuracy (%): 61.01 -> 47.48 | Hurt |
| Combine Manhattan and Euclidean movement costs (ablation) | STORM cost function · VISOR (SD 1.4) | Object Accuracy (%): 61.01 -> 46.60 | Hurt |
| Add overlap prevention to ST Cost (ablation) | STORM · VISOR (SD 1.4) | Object Accuracy (%): 31.38 -> 61.01 | Helped |
| Use STORM instead of REVISION | Stable Diffusion 1.4 · VISOR uncond (SD 1.4) | VISOR (%): 52.71 -> 57.58 | Helped |
| Use STORM instead of REVISION | Stable Diffusion 2.1 · VISOR uncond (SD 2.1) | VISOR (%): 47.11 -> 59.35 | Helped |
| Use STORM instead of Stable Diffusion | Stable Diffusion 2.1 · T2I-CompBench spatial relationship | Spatial score: 0.1342 -> 0.1981 | Helped |
Surprising findings
- Applying spatial optimization only in early denoising steps (timesteps 1-24 vs later ranges) gives the largest gains, and resolving mislocated objects also reduced missing-object errors.
Limitations
- Relies on cross-attention maps and centroid references, so gains depend on attention quality; overall OA remains around 61% on VISOR.
- Evaluation of object and attribute consistency uses CLIP and TIFA metrics whose reliability is limited; user study uses only 10 custom prompts.
Experimental setup
Datasets: VISOR T2I-CompBench
Benchmarks: VISOR T2I-CompBench TIFA CLIP score
Metrics: VISOR uncond/cond accuracy Object Accuracy (OA) BLIP-VQA attribute binding UniDet spatial score CLIP image-text similarity TIFA score user study accuracy
Baselines: Stable Diffusion 1.4 Stable Diffusion 2.1 Attend-and-Excite Divide-and-Bind INITNO CONFORM Structure Diffusion Layout Guidance REVISION SPRIGHT Composable Diffusion Control-GPT
Builds on: Attend-and-Excite Sinkhorn algorithm Stable Diffusion Divide-and-Bind INITNO
Closest papers by meaning
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.89)
- Grounded Text-to-Image Synthesis with Attention Refocusing (CVPR 2024 · similarity 0.89)
- SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-Form Layout-to-Image Generation (AAAI 2024 · similarity 0.89)
- Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models (AAAI 2024 · similarity 0.88)
- Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.88)
- Control and Realism: Best of Both Worlds in Layout-to-Image without Training (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.