Research map / Image generation / Text-to-image diffusion
Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Probes cross- and self-attention maps in Stable Diffusion and proposes Free-Prompt-Editing (FPE), a tuning-free image editing method that only replaces self-attention maps in layers 4-14 during denoising.
Problem
Tuning-free text-guided image editing with Stable Diffusion relies on cross-attention map replacement, which is unstable and requires source prompts; little is known about what attention layers encode.
Approach
Probing classifiers show cross-attention maps encode object/attribute category information (causing edit failures), while self-attention maps encode spatial structure; so editing only self-attention maps in layers 4-14, without a source prompt, preserves layout while enabling edits.
Key results
- On Wild-TI2I and ImageNet-R-TI2I, FPE achieves the highest CDS on most splits (e.g., 0.3559 on ImageNet-R-TI2I fake vs. 0.3394 for P2P).
- FPE takes about 6.30 s per generated image on an A100 versus 335.65 s for PnP.
- On Car-real-edit, FPE improves CDS from 0.2288 (P2P) to 0.2605.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FPE instead of P2P | Stable Diffusion 1.5 image editing · Car-fake-edit | Clip Directional Similarity: 0.2451 -> 0.2659 | Helped |
| Use FPE instead of P2P | Stable Diffusion 1.5 image editing · Car-real-edit | Clip Directional Similarity: 0.2288 -> 0.2605 | Helped |
| Use FPE instead of P2P | Stable Diffusion 1.5 image editing · ImageNet-fake-edit | Clip Directional Similarity: 0.2401 -> 0.2560 | Helped |
| Use FPE instead of P2P | Stable Diffusion 1.5 image editing · ImageNet-real-edit | Clip Directional Similarity: 0.2426 -> 0.2468 | Helped |
| Replace cross-attention maps during editing (ablation) | Diffusion image editing · Image editing examples in Figure 4 | Editing success | Hurt |
| Replace self-attention maps in all layers (ablation) | Diffusion image editing · Image editing examples in Figure 4 | Target editing success | Hurt |
| Replace self-attention maps in Layers 4 to 14 (ablation) | FPE · Image editing examples in Figure 4 | Structure preservation and editing success | Helped |
| Leave the self-attention map unaltered (ablation) | Diffusion image editing · Image editing examples in Figure 4 | Source structure preservation | Hurt |
| Use FPE instead of the strongest baseline, MasaCtrl | Stable Diffusion 1.5 image editing · ImageNet-R-TI2I fake | Clip Directional Similarity: 0.3024 -> 0.3559 | Helped |
| Use FPE instead of the strongest baseline, P2P | Stable Diffusion 1.5 image editing · ImageNet-R-TI2I real | Clip Directional Similarity: 0.2146 -> 0.2271 | Helped |
| Use FPE instead of the strongest baseline, MasaCtrl | Stable Diffusion 1.5 image editing · Wild-TI2I fake | Clip Directional Similarity: 0.3474 -> 0.3116 | Hurt |
| Use FPE instead of the strongest baseline, P2P | Stable Diffusion 1.5 image editing · Wild-TI2I real | Clip Directional Similarity: 0.1930 -> 0.2234 | Helped |
Surprising findings
- Replacing cross-attention maps for all layers yields the least satisfactory edits, and editing cross-attention is dispensable for TIE.
- Cross-attention maps classify color and animal categories with high accuracy (e.g., 98% for sheep), showing they carry semantic attribute information beyond weights.
Limitations
- Editing is bounded by the generative capability of the underlying TIS model.
- Real image editing requires reconstruction via VQ autoencoder, which can lose facial details.
Experimental setup
Datasets: Wild-TI2I ImageNet-R-TI2I FlexIT prompts ImageNet Stanford Cars (CARS196)
Benchmarks: Wild-TI2I ImageNet-R-TI2I Car-fake-edit ImageNet-fake-edit Car-real-edit ImageNet-real-edit
Metrics: CLIP score CLIP directional similarity editing time per image
Baselines: P2P PnP SDEdit DiffEdit MasaCtrl Pix2pix-zero Shape-guided InstructPix2Pix
Builds on: Prompt-to-Prompt Plug-and-Play Null-text inversion SDEdit DDIM inversion
Closest papers by meaning
- Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjects (CVPR 2025 · similarity 0.91)
- Stable Flow: Vital Layers for Training-Free Image Editing (CVPR 2025 · similarity 0.90)
- Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks (AAAI 2024 · similarity 0.90)
- Grounded Text-to-Image Synthesis with Attention Refocusing (CVPR 2024 · similarity 0.90)
- Devil is in the Detail: Towards Injecting Fine Details of Image Prompt in Image Generation via Conflict-free Guidance and Stratified Attention (CVPR 2025 · similarity 0.89)
- Localizing and Editing Knowledge In Text-to-Image Generative Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.