Research map / Image generation / Editing and style transfer
LightningDrag: Lightning Fast and Accurate Drag-based Image Editing Emerging from Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Redefines drag-based image editing as conditional generation on Stable Diffusion inpainting, trained on video frame pairs, achieving ~1 s edits with better accuracy than prior diffusion-based drag methods.
Problem
Drag-based image editing is slow (often over a minute per edit) and has low success rates, since prior GAN and diffusion methods rely on latent optimization or gradient-based guidance at inference.
Approach
Treat drag editing as a single-pass conditional generation task: an SD inpainting U-Net conditioned on source-image appearance features (reference-only encoder), a point embedding network for handle/target point maps injected into self-attention, trained on automatically mined video frame pairs with CoTracker2 correspondences; plus test-time noise priors and point-following CFG with decaying sched
Key results
- On DragBench, LightningDrag achieves the lowest Mean Distance (18.62) versus 28.58 for DiffEditor and 33.16 for DragDiffusion, with IF 0.885.
- LightningDrag takes 2.06 s per edit on an A100 and 0.92 s with LCM-LoRA, versus 55 s for DragDiffusion and 98 s for Readout Guidance.
- SDEDrag attains the highest IF (0.921) but has MD 45.78, attributed to near-identity mappings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LightningDrag instead of DiffEditor | Drag-based image editing on DragBench · DragBench | MD: 28.579 -> 18.62 | Helped |
| Use LightningDrag instead of SDEDrag | Drag-based image editing on DragBench · DragBench | IF: 0.921 -> 0.885 | Hurt |
| Add LCM-LoRA acceleration | LightningDrag · DragBench | IF: 0.885 -> 0.890 | Helped |
| Add LCM-LoRA acceleration | LightningDrag · DragBench | MD: 18.62 -> 18.95 | Hurt |
| Add LCM-LoRA acceleration | LightningDrag · DragBench square images, 512×512, NVIDIA A100 | time (s): 2.06_{± 0.05} -> 0.92_{± 0.02} | Helped |
| Use noised source latents instead of pure random noise, mixed noise latents, or copy-and-paste noise latents (ablation) | LightningDrag · Drag-based image editing | editing quality | Helped |
| Add point-following classifier-free guidance rather than using no CFG (ablation) | LightningDrag · Drag-based image editing | point-following success | Helped |
| Use constant CFG rather than no CFG (ablation) | LightningDrag · Drag-based image editing | point-following success | Helped |
| Use constant CFG rather than a decaying CFG schedule (ablation) | LightningDrag · Drag-based image editing | image quality | Hurt |
| Use inverse-square CFG decay rather than linear or square decay (ablation) | LightningDrag · Drag-based image editing | image quality | Helped |
| Use linear or square CFG decay rather than inverse-square decay (ablation) | LightningDrag · Drag-based image editing | image quality | Hurt |
| Add more handle-target point pairs to the drag instruction (ablation) | LightningDrag · Drag-based image editing | editing quality | Helped |
Surprising findings
- SDEDrag's highest image fidelity largely reflects undesired identity mapping rather than successful dragging.
Limitations
- Built on Stable Diffusion V1.5, so small regions such as hands and faces lack detail
- Training uses an internal, non-released video dataset
- Local deformations not present in videos are only generalized empirically, not guaranteed
Experimental setup
Datasets: internal video dataset (220k sampled pairs) DragBench
Benchmarks: DragBench
Metrics: Image Fidelity (1-LPIPS) Mean Distance inference time (s)
Baselines: DragGAN DragDiffusion DiffEditor Readout Guidance SDEDrag
Builds on: DragGAN DragDiffusion Stable Diffusion inpainting IP-Adapter CoTracker2
Closest papers by meaning
- FastDrag: Manipulate Anything in One Step (NeurIPS 2024 · similarity 0.91)
- FlowDrag: 3D-aware Drag-based Image Editing with Mesh-guided Deformation Vector Flow Fields (ICML 2025 · similarity 0.89)
- CLIPDrag: Combining Text-based and Drag-based Instructions for Image Editing (ICLR 2025 · similarity 0.89)
- GoodDrag: Towards Good Practices for Drag Editing with Diffusion Models (ICLR 2025 · similarity 0.88)
- EasyDrag: Efficient Point-based Manipulation on Diffusion Models (CVPR 2024 · similarity 0.88)
- DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.