Research map / Image generation / Editing and style transfer
DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends DragGAN-style interactive point-based editing to Stable Diffusion by optimizing a single-timestep DDIM latent, with LoRA identity-preserving fine-tuning and reference-latent control, and introduces the DragBench benchmark.
Problem
Interactive point-based (drag) image editing is limited to GAN generators whose capacity restricts generality, while diffusion-based editing mostly relies on text prompts and lacks precise spatial control.
Approach
Optimize only the diffusion latent at one selected DDIM inversion step (t=35 of 50) using UNet features for motion supervision and point tracking, then denoise with self-attention key/value injection from the original image to preserve identity; LoRA fine-tuning on the input image further improves consistency.
Key results
- DragDiffusion outperforms DragGAN on DragBench in both Image Fidelity and Mean Distance (Fig. 8, values shown graphically).
- Optimizing the latent at inversion step t=35 of 50 gives the lowest Mean Distance while keeping a decent Image Fidelity.
- Identity-preserving LoRA fine-tuning for 80 steps (about 25 seconds on an A100) sharply reduces Mean Distance; removing it causes a catastrophic MD increase.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DragDiffusion instead of DragGAN | DRAGBENCH editing · DRAGBENCH | Image Fidelity (1-LPIPS) | Helped |
| use DragDiffusion instead of DragGAN | DRAGBENCH editing · DRAGBENCH | Mean Distance | Helped |
| add identity-preserving fine-tuning (ablation) | DragDiffusion · DRAGBENCH | Mean Distance | Helped |
| add reference-latent-control (ablation) | DragDiffusion · DRAGBENCH | Image Fidelity (1-LPIPS) | Helped |
| increase identity-preserving fine-tuning steps from 0 toward 80 (ablation) | DragDiffusion · DRAGBENCH | Image Fidelity (1-LPIPS) | Helped |
| increase identity-preserving fine-tuning steps from 0 toward 80 (ablation) | DragDiffusion · DRAGBENCH | Mean Distance | Helped |
| extend identity-preserving fine-tuning beyond 80 steps (ablation) | DragDiffusion · DRAGBENCH | Image Fidelity and Mean Distance | No clear effect |
| increase the number of inversion steps (ablation) | DragDiffusion · DRAGBENCH | Image Fidelity (1-LPIPS) | Hurt |
| vary the number of inversion steps (ablation) | DragDiffusion · DRAGBENCH | Mean Distance | Mixed |
| use deeper UNet decoder feature blocks (ablation) | DragDiffusion · DRAGBENCH | Image Fidelity (1-LPIPS) | Helped |
| vary the UNet decoder feature block (ablation) | DragDiffusion · DRAGBENCH | Mean Distance | Mixed |
Surprising findings
- Fine-tuning LoRA for only 80 steps suffices, versus 1000 steps typical for subject-driven generation.
Limitations
- Relies on DragGAN-style motion supervision and point tracking, which can be slow or inaccurate for large moves (limitations in Appendix E, not included in text)
- Evaluation uses LPIPS-based fidelity and DIFT-based point tracking, which are proxy metrics
- Baseline DragGAN requires per-class StyleGAN selection via CLIP ensemble
Experimental setup
Datasets: DragBench
Benchmarks: DragBench
Metrics: Image Fidelity (1-LPIPS) Mean Distance
Baselines: DragGAN
Builds on: DragGAN DragDiffusion-related DDIM inversion Plug-and-Play (self-attention feature injection) LoRA DIFT
Closest papers by meaning
- DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models (ICLR 2024 · similarity 0.92)
- EasyDrag: Efficient Point-based Manipulation on Diffusion Models (CVPR 2024 · similarity 0.91)
- Drag Your Noise: Interactive Point-based Editing via Diffusion Semantic Propagation (CVPR 2024 · similarity 0.91)
- FastDrag: Manipulate Anything in One Step (NeurIPS 2024 · similarity 0.89)
- 3DGS-Drag: Dragging Gaussians for Intuitive Point-Based 3D Editing (ICLR 2025 · similarity 0.89)
- Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.