Research map / Image generation / Editing and style transfer
Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InstDiffEdit, a training-free method that derives attention-based editing masks at each denoising step of Stable Diffusion for instant, automatic, mask-guided image editing, and introduces the Editing-Mask benchmark.
Problem
Diffusion-based image editing typically needs manual or offline semantic masks, which makes it slow and labor-intensive; fully automatic mask generation (e.g., DiffEdit) is also time-consuming.
Approach
Use the cross-attention maps of a text-to-image diffusion model at each denoising step to produce masks on the fly: the start-token attention is used to locate the target token, similar tokens are aggregated with thresholded cosine weights, and the resulting mask guides denoising and a final inpainting pass, with no training.
Key results
- InstDiffEdit reaches 56.2% IOU on Editing-Mask versus 33.0% for DiffEdit, with a changing rate of 3.71 vs 2.45.
- InstDiffEdit runs in 10.8 s per image versus 64.0 s for DiffEdit, about 5 to 6 times faster as claimed.
- On the Imagen set, InstDiffEdit improves FID to 55.3 from 58.8 for DiffEdit and LPIPS to 17.0 from 29.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace DiffEdit with InstDiffEdit | Stable Diffusion v1.4 image editing · Editing-Mask | IOU: 33.0 -> 56.2 | Helped |
| replace DiffEdit with InstDiffEdit | Stable Diffusion v1.4 image editing · Editing-Mask | time: 64.0 -> 10.8 | Helped |
| replace DiffEdit with InstDiffEdit | Stable Diffusion v1.4 image editing · ImageNet | LPIPS: 27.9 -> 28.6 | Hurt |
| replace DiffEdit with InstDiffEdit | Stable Diffusion v1.4 image editing · ImageNet | CSFID: 70.9 -> 65.1 | Helped |
| replace DiffEdit with InstDiffEdit | Stable Diffusion v1.4 image editing · Imagen | LPIPS: 29.7 -> 17.0 | Helped |
| replace DiffEdit with InstDiffEdit | Stable Diffusion v1.4 image editing · Imagen | CLIPScore: 0.247 -> 0.249 | Helped |
| add mask guidance instead of editing without a mask (ablation) | InstDiffEdit · Editing-Mask | editing performance | Helped |
| increase noise strength r from 0.4 to 0.6 (ablation) | InstDiffEdit · Editing-Mask | IOU: 52.9 -> 49.6 (φ=0.1); 55.7 -> 54.6 (φ=0.2); 52.0 -> 54.2 (φ=0.3) | Mixed |
| increase threshold φ from 0.1 to 0.3 (ablation) | InstDiffEdit · Editing-Mask | IOU: 51.9 -> 54.3 (r=0.5) | Mixed |
Surprising findings
- Mask-free attention and latent editing baselines change the background heavily, showing that masks are essential for background preservation.
- On ImageNet, InstDiffEdit has slightly worse LPIPS than DiffEdit (28.6 vs 27.9), while PnP has the best LPIPS (27.3).
Limitations
- DiffEdit is a non-official re-implementation, so its numbers may not reflect the original method.
- Editing-Mask is small (200 images) and relies on human-labeled masks; the Imagen set is generated by a closed model.
- Method relies on hand-set thresholds (r, phi, gamma1, gamma2) tuned on the evaluation benchmark.
Experimental setup
Datasets: ImageNet Imagen-generated images
Benchmarks: Editing-Mask ImageNet (FlexIT setting) Imagen-generated editing set
Metrics: IOU C_m C_non changing rate LPIPS CSFID FID CLIPScore editing time
Baselines: SDEdit CycleDiffusion PtP PnP DiffEdit
Builds on: Stable Diffusion DiffEdit Prompt-to-Prompt Blended Diffusion RePaint
Closest papers by meaning
- Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.90)
- Accelerating Text-to-Image Editing via Cache-Enabled Sparse Diffusion Inference (AAAI 2024 · similarity 0.90)
- Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer (NeurIPS 2025 · similarity 0.89)
- BARET: Balanced Attention Based Real Image Editing Driven by Target-Text Inversion (AAAI 2024 · similarity 0.89)
- Stable Flow: Vital Layers for Training-Free Image Editing (CVPR 2025 · similarity 0.89)
- LEDITS++: Limitless Image Editing using Text-to-Image Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.