Research map / Image generation / Editing and style transfer
An Item Is Worth a Prompt: Versatile Image Editing with Disentangled Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D-Edit, which segments an image into items, assigns each item a learned prompt via two-step finetuning, and uses grouped cross-attention to enable disentangled item-level editing (text, image, mask, removal).
Problem
Prompt-based editing of diffusion models is unstable because pretrained DPMs are trained on captions, so editing words changes the whole image; spatial masks are often ignored by DPMs, causing disharmonious edits.
Approach
Segment the image into items, give each item its own learned set of random-init prompt tokens, and replace full cross-attention with grouped cross-attention so each item attends only to its own prompt; two-step optimization (text embeddings, then cross-attention/LoRA weights) builds item-prompt associations, enabling editing by changing prompts, masks, or mappings.
Key results
- Text-guided editing LPIPS 0.179 and CLIP-T 42.0 vs P2P 0.401/39.5 and SDEdit 0.432/32.1 on D-Item(Text).
- Image-guided editing LPIPS_t 0.340 and CLIP-I 66.4, outperforming AnyDoor (0.608, 60.2) and Paint-by-Example (0.465, 50.8).
- Item removal user study: quality 4.01 and fidelity 4.44 for D-Edit vs 3.26 and 2.42 for SDXL-inpaint.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use D-Edit instead of P2P | D-Edit · D-Item(Text) | LPIPS: 0.401 -> 0.179 | Helped |
| Use D-Edit instead of P2P | D-Edit · D-Item(Text) | CLIP-T: 39.5 -> 42.0 | Helped |
| Use D-Edit instead of Paint-by-Example | D-Edit · D-Item(Image) | LPIPS_t: 0.465 -> 0.340 | Helped |
| Use D-Edit instead of Paint-by-Example | D-Edit · D-Item(Image) | LPIPS_r: 0.833 -> 0.701 | Helped |
| Use D-Edit instead of Anydoor | D-Edit · D-Item(Image) | CLIP-I: 60.2 -> 66.4 | Helped |
| Use D-Edit instead of SDXL-inpaint | D-Edit · Item removal user study | Quality: 3.26 -> 4.01 | Helped |
| Use D-Edit instead of SDXL-inpaint | D-Edit · Item removal user study | Fidelity: 2.42 -> 4.44 | Helped |
| Use 10 rather than 5 tokens per item (ablation) | D-Edit · D-Item(Text) | LPIPS: 0.179 -> 0.413 | Hurt |
| Use 10 rather than 5 tokens per item (ablation) | D-Edit · D-Item(Text) | CLIP-T: 42.0 -> 30.1 | Hurt |
| Add cross-attention disentanglement (ablation) | D-Edit · Text-guided editing | preservation of original item information | Helped |
Limitations
- Color is harder to learn than shape in image-based editing
- Out-of-domain or blurry reference items (e.g., celebrity faces, blurred content) are hard to reconstruct
- Requires per-image finetuning of the diffusion model for each input image
Experimental setup
Datasets: D-Item(Text) (100 multi-item images, self-constructed) D-Item(Image) (400 item-item pairs, derived)
Benchmarks: D-Item(Text) D-Item(Image)
Metrics: LPIPS CLIP-T CLIP-I user study quality/fidelity score
Baselines: P2P with null-text inversion SDEdit AnyDoor Paint-by-Example TF-ICON SDXL-inpaint
Builds on: Textual Inversion DreamBooth Prompt-to-Prompt SDEdit LoRA
Closest papers by meaning
- Textualize Visual Prompt for Image Editing via Diffusion Bridge (AAAI 2025 · similarity 0.90)
- RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformers (NeurIPS 2025 · similarity 0.89)
- PAIR Diffusion: A Comprehensive Multimodal Object-Level Image Editor (CVPR 2024 · similarity 0.89)
- Prompt Augmentation for Self-supervised Text-guided Image Manipulation (CVPR 2024 · similarity 0.89)
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.89)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.