Research map / Image generation / Editing and style transfer
CLIPDrag: Combining Text-based and Drag-based Instructions for Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIPDrag, which combines text prompts (via CLIP global gradients) with drag points (via motion supervision) on a LoRA-finetuned Stable Diffusion 1.5 to perform text-drag image editing with less ambiguity and faster convergence.
Problem
Text-based image editing lacks precise spatial/fine-grained control, while drag-based editing is ambiguous since the same handle-target pairs can correspond to multiple semantic edits and optimization is slow.
Approach
Use CLIP text guidance as a global gradient and drag motion supervision as a local gradient, decompose the global gradient into identity and edit components relative to the local gradient for fusion, and use a fast point tracking that only considers candidates closer to targets.
Key results
- On DragBench drag-only editing, CLIPDrag achieves mean distance 32.3 vs 35.1 for DragDiffusion at 160 max iterations, with higher 1-LPIPS at most step settings.
- Average inference time on a single 3090 is 47.8s for CLIPDrag versus 80.3s for DragDiffusion and 69.4s for FreeDrag.
- Ablation shows simple gradient addition fails to resolve ambiguity, and ProGrad preserves fidelity poorly compared with the proposed identity/edit decomposition.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add text guidance alongside drag points (ablation) | CLIPDrag · DRAGBENCH text-drag editing | editing ambiguity | Helped |
| Replace global-local gradient fusion with simple gradient addition (ablation) | CLIPDrag · Text-drag editing | ambiguity reduction | Hurt |
| Replace global-local gradient fusion with ProGrad, omitting the identity component (ablation) | CLIPDrag · Text-drag editing | image fidelity | Hurt |
| Add the identity component in global-local gradient fusion (ablation) | CLIPDrag · Text-drag editing | image fidelity | Helped |
| Replace fast point tracking with normal DragDiff point tracking (ablation) | CLIPDrag · DRAGBENCH drag-based editing | optimization iterations | Hurt |
| Use fast point tracking instead of normal point tracking (ablation) | CLIPDrag · DRAGBENCH drag-based editing | image quality | No clear effect |
| Add Fast Point Tracking (ablation) | CLIPDrag · DRAGBENCH drag-based editing | optimization speed | Helped |
| Use CLIPDrag instead of DragDiff | DRAGBENCH drag-based editing, max iterations 20 · DRAGBENCH | mean distance: 50.6 -> 45.1 | Helped |
| Use CLIPDrag instead of DragDiff | DRAGBENCH drag-based editing, max iterations 80 · DRAGBENCH | image fidelity (IF): 0.87 -> 0.90 | Helped |
| Use CLIPDrag rather than prior methods | Image editing inference · DragBench examples | inference time: 80.3s -> 47.8s | Helped |
Surprising findings
- Simply adding the CLIP global gradient to the drag gradient does not alleviate ambiguity because CLIP guidance has little effect at a single fixed diffusion step.
Limitations
- Text-drag results are evaluated mainly qualitatively; quantitative text-drag evaluation is not reported.
- Conflicting text and drag instructions can yield ambiguous or implausible edits, especially at high text strength (lambda=100).
- Text-drag editing requires specifying both edit prompts and drag points, adding user input.
Experimental setup
Datasets: DragBench
Benchmarks: DragBench
Metrics: mean distance image fidelity (1-LPIPS) inference time
Baselines: DragDiffusion FreeDrag RegionDrag StableDrag InstantDrag LightningDrag DiffCLIP FastDrag SDE-Drag
Builds on: DragDiffusion DragGAN ProGrad LoRA DDIM inversion
Closest papers by meaning
- LightningDrag: Lightning Fast and Accurate Drag-based Image Editing Emerging from Videos (ICML 2025 · similarity 0.89)
- Localize, Understand, Collaborate: Semantic-Aware Dragging via Intention Reasoner (NeurIPS 2024 · similarity 0.89)
- FastDrag: Manipulate Anything in One Step (NeurIPS 2024 · similarity 0.88)
- GoodDrag: Towards Good Practices for Drag Editing with Diffusion Models (ICLR 2025 · similarity 0.88)
- FlowDrag: 3D-aware Drag-based Image Editing with Mesh-guided Deformation Vector Flow Fields (ICML 2025 · similarity 0.87)
- DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.