Research map / Image generation / Editing and style transfer
DesignEdit: Unify Spatial-Aware Image Editing via Training-free Inpainting with a Multi-Layered Latent Diffusion Framework
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DesignEdit, a training-free multi-layered latent diffusion framework on SDXL-1.0 that inpaints the background via key-masking self-attention and fuses independently edited element layers to unify spatial-aware image editing operations.
Problem
Spatial-aware image editing (moving, resizing, removing, adding elements) struggles with background filling after removal, preserving object identity, and performing complex multi-object edits in a single pass.
Approach
Mask the Key (not Query or Value) in self-attention during the first K denoising steps so that the background latent inpaints removed regions training-free, plus artifact suppression with a GPT-4V-proposed refinement mask; then decompose the image into per-element latent layers and paste them onto a canvas latent at timestep T-K following GPT-4V-generated layer-wise instructions.
Key results
- On MagicBrush removal (51 examples), the training-free method reaches FID 38.883 and DINO 0.971, comparable to SDXL-inpainting (FID 44.621, DINO 0.959), while LaMa is best overall (FID 25.077).
- A user study with 452 votes from 113 participants gave the method a 51% preference rate on object removal against inpainting baselines.
- In a single-object spatial editing user study (73 participants, 1,460 votes, 10 examples), the method was compared against Self-Guidance and DiffEditor on image quality and edit accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the [50 ∼ 10] key-masking range rather than applying it across [50 ∼ 0] (ablation) | DesignEdit · Image removal | removal quality | Helped |
| Mask the Query instead of the Key (ablation) | DesignEdit · Image removal | pixel clarity | Hurt |
| Mask the Value instead of the Key (ablation) | DesignEdit · Image removal | image coherence | Hurt |
| Use key masking instead of Query or Value masking (ablation) | DesignEdit · Image removal | background integration | Helped |
| Use DesignEdit instead of LaMa for mask-guided removal | SDXL-1.0 · MagicBrush test set object removal | user preference rate: 51% | Helped |
Surprising findings
- Training-free key-masking self-attention achieves removal quality comparable to fine-tuned inpainting models such as SDXL-inpainting.
Limitations
- GPT-4V layer planning can misestimate object size and position or set incorrect layer order, causing failures in complex cross-image editing.
- Removal quality is only comparable to, not better than, fine-tuned inpainting; LaMa outperforms it on most metrics.
- Evaluation of removal is limited to 51 MagicBrush examples and spatial editing to 10 examples.
Experimental setup
Datasets: MagicBrush
Benchmarks: MagicBrush
Metrics: L1 L2 CLIP-I DINO CLIP-T FID LPIPS user preference win rate
Baselines: LaMa ControlNet-inpainting SDXL-inpainting Uni-paint Self-Guidance DiffEditor
Builds on: Stable Diffusion XL Prompt-to-Prompt MasaCtrl DDIM
Closest papers by meaning
- PixPerfect: Seamless Latent Diffusion Local Editing with Discriminative Pixel-Space Refinement (NeurIPS 2025 · similarity 0.89)
- Move Anything with Layered Scene Diffusion (CVPR 2024 · similarity 0.89)
- RAD: Region-Aware Diffusion Models for Image Inpainting (CVPR 2025 · similarity 0.88)
- Towards Enhanced Image Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency (CVPR 2025 · similarity 0.88)
- In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment (NeurIPS 2024 · similarity 0.88)
- Paint by Inpaint: Learning to Add Image Objects by Removing Them First (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.