Research map / Image generation / Text-to-image diffusion
Directed Diffusion: Direct Control of Object Placement through Attention Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free method (Directed Diffusion) that steers Stable Diffusion object placement by optimizing a small weight vector over trailing cross-attention maps so prompt-word attention concentrates inside user-given bounding boxes.
Problem
Text-to-image diffusion models struggle to compose multiple objects in specified positional relationships, which limits controllable storytelling and scene composition.
Approach
Edit cross-attention maps of directed prompt words during early denoising steps by injecting Gaussian activation inside user bounding boxes and optimizing a small weight vector over trailing (non-prompt) attention maps so the effect propagates through the network, requiring no training and only a few lines of code.
Key results
- DD achieves CLIP score 0.824 on SceneComp vs 0.821 for SD, 0.802 for GLIGEN and 0.791 for CD.
- On TwoMasks DD scores 0.851 vs 0.842 for SD and 0.802 for GLIGEN.
- On OneMask DD scores 0.807, below GLIGEN (0.810) and SD (0.834).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Directed Diffusion instead of the strongest listed baseline, SD | Scene compositing · SceneComp | CLIP score: 0.821 -> 0.824 | Helped |
| Use Directed Diffusion instead of the strongest listed baseline, SD | One-object placement · OneMask | CLIP score: 0.834 -> 0.807 | Hurt |
| Use Directed Diffusion instead of the strongest listed baseline, SD | Two-object placement · TwoMasks | CLIP score: 0.842 -> 0.851 | Helped |
| Remove optimization and directly inject Gaussian-falloff activations into directed-word and trailing attention maps (ablation) | Directed Diffusion · Object placement and scene generation | placement and scene quality | Mixed |
| Use only a small number of edited trailing attention maps (ablation) | Direct-injection ablation · Object placement and scene generation | semantic and placement quality | Hurt |
| Use too many edited trailing attention maps (ablation) | Direct-injection ablation · Object placement and scene generation | background preservation | Hurt |
| Use an intermediate number of edited trailing attention maps and editing steps (ablation) | Direct-injection ablation · Object placement and scene generation | semantic and placement quality | Helped |
| Vary the number of edited trailing attention maps across 5, 10, 15, and 20 (ablation) | Direct-injection ablation · A photo of a squirrel eating a burger | semantic and placement quality | Mixed |
| Vary attention editing steps across 1, 3, 5, 10, and 15 (ablation) | Direct-injection ablation · A photo of a squirrel eating a burger | semantic and placement quality | Mixed |
Surprising findings
- Removing the optimization and directly injecting Gaussian activations into cross-attention maps was unstable, requiring grid search; the authors found that the CLIP score may be blind to the improvement achieved on multi-object composition.
- Higher IoU with bounding boxes was found to interfere with visual quality, since natural object-object interaction (e.g., occlusion) lowers IoU.
Limitations
- Some random seeds fail to produce the desired subject or produce distorted objects with missing limbs, requiring trial-and-error.
- Objects do not strictly respect the bounding box and may extend outside it; number of editing steps must be specified.
- Multi-object results are unreliable with more than two bounding boxes without additional compositing; CLIP scores and IoU are poor proxies for the task.
Experimental setup
Benchmarks: SceneComp OneMask TwoMasks
Metrics: CLIP score IoU
Baselines: Stable Diffusion GLIGEN BoxDiff Composable Diffusion Blended Latent Diffusion MultiDiffusion
Builds on: Stable Diffusion Prompt-to-prompt MagicMix Blended Latent Diffusion SDEdit
Closest papers by meaning
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- Multi-Region Text-Driven Manipulation of Diffusion Imagery (AAAI 2024 · similarity 0.89)
- Grounded Text-to-Image Synthesis with Attention Refocusing (CVPR 2024 · similarity 0.88)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.88)
- DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.