Research map / Image generation / Editing and style transfer
DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DragonDiffusion, a tuning-free drag-style image editing method that turns diffusion-feature correspondence into energy-based gradient guidance plus a memory-bank visual cross-attention for consistency.
Problem
Precise, fine-grained image editing (drag-style moving, resizing, dragging, appearance replacing, pasting) is hard with text-conditioned diffusion models, which lack a compact editable latent space, and prior GAN-based drag methods are limited in generation quality and generalization.
Approach
Reuses the pre-trained SD UNet decoder features as a tuning-free energy function measuring feature correspondence between source and target regions, converting editing operations into gradient guidance during DDIM sampling; a memory bank from DDIM inversion supplies keys/values for visual cross-attention to preserve content consistency.
Key results
- On 17/68-point face manipulation, DragonDiffusion reaches FID 35.75/34.58, the best among compared methods, versus 38.06/36.55 for DragDiffusion.
- Preparing stage takes 3.62s versus 48.25s for DragDiffusion and 52.40s for DragGAN on 1-point dragging at 512x512 on an A100.
- DragGAN achieves lower keypoint distance (15.96 / 10.60) than DragonDiffusion (18.51 / 13.94), so DragonDiffusion trades some accuracy for consistency and quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DragonDiffusion instead of DragDiffusion | CelebA-HQ face manipulation · 17-point manipulation | point distance: 22.95 -> 18.51 | Helped |
| Use DragonDiffusion instead of DragDiffusion | CelebA-HQ face manipulation · 68-point manipulation | point distance: 17.32 -> 13.94 | Helped |
| Use DragonDiffusion instead of DragDiffusion | CelebA-HQ face manipulation · 17/68-point manipulation | FID: 38.06/36.55 -> 35.75/34.58 | Helped |
| Use DragonDiffusion instead of DragGAN | CelebA-HQ face manipulation · 17-point manipulation | point distance: 15.96 -> 18.51 | Hurt |
| Use DragonDiffusion instead of DragGAN | CelebA-HQ face manipulation · 68-point manipulation | point distance: 10.60 -> 13.94 | Hurt |
| Use DragonDiffusion instead of DragGAN | CelebA-HQ face manipulation · 17/68-point manipulation | FID: 39.27/39.50 -> 35.75/34.58 | Helped |
| Replace the DDIM inversion prior with random initialization of z_T (ablation) | DragonDiffusion object moving · Object moving | content consistency | Hurt |
| Remove content consistency guidance (ablation) | DragonDiffusion object moving · Object moving | local distortion | Hurt |
| Remove visual cross-attention (ablation) | DragonDiffusion object moving · Object moving | content consistency | Hurt |
| Use a separate generation branch for guidance instead of the memory bank (ablation) | DragonDiffusion editing · Image editing | editing quality | Hurt |
| Add null-text optimization to a separate guidance branch (ablation) | DragonDiffusion editing · Image editing | guidance accuracy | Helped |
Surprising findings
- DragGAN is more accurate on keypoint dragging than DragonDiffusion, yet DragonDiffusion scores better on FID and user-study preferences.
Limitations
- Gradient guidance is applied only in the first 30 of 50 steps, with hyperparameters (weights, learning rate, guidance layers) set heuristically per task.
- Keypoint accuracy lags DragGAN, and inference (15.93s) is slower than DragGAN's inference stage.
- Relies on the generation prior of Stable Diffusion v1.5, so output quality is bounded by the base model.
Experimental setup
Datasets: CelebA-HQ
Benchmarks: CelebA-HQ face keypoint manipulation (17 and 68 points)
Metrics: Euclidean distance to target points FID inference and preparing time user study votes
Baselines: DragGAN DragDiffusion UserControllableLT
Builds on: Stable Diffusion DIFT DDIM inversion classifier guidance DragGAN
Closest papers by meaning
- DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing (CVPR 2024 · similarity 0.92)
- FastDrag: Manipulate Anything in One Step (NeurIPS 2024 · similarity 0.89)
- Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators (ICLR 2024 · similarity 0.88)
- Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.88)
- A Diffusion-Based Framework for Occluded Object Movement (AAAI 2025 · similarity 0.88)
- Multi-Region Text-Driven Manipulation of Diffusion Imagery (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.