Research map / Image generation / Editing and style transfer
Drag Your Noise: Interactive Point-based Editing via Diffusion Semantic Propagation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DragNoise, which performs point-based image editing by optimizing the U-Net bottleneck feature at one early timestep and propagating it to later denoising steps, avoiding latent-map retracing.
Problem
Point-based interactive image editing with diffusion models: DragDiffusion's optimization of the noisy latent map causes gradient vanishing, under-dragging, and poor reconstruction fidelity.
Approach
Treat the U-Net predicted noise as a semantic editor: optimize the bottleneck feature at a single early timestep (t=35) with a semantic alignment loss, then substitute the optimized bottleneck feature into subsequent denoising steps, since high-level semantics are established early and stable.
Key results
- On DragBench, DragNoise achieves better dragging accuracy (mean distance) and image fidelity than DragGAN, FreeDrag and DragDiffusion.
- Optimization converges in about 25 steps versus 56 for DragDiffusion, cutting optimization time by over 50% (about 10 s vs over 22 s per 512x512 image on a Tesla V100).
- Ablation shows optimizing the bottleneck feature gives the best quality and controllability compared with encoder or decoder blocks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use bottleneck-feature optimization and semantic propagation instead of DragDiffusion latent-map optimization | DragNoise vs. DragDiffusion · 512×512 image editing | optimization time: over 50% | Helped |
| use bottleneck-feature optimization rather than latent-map optimization | DragNoise vs. DragDiffusion · optimization example | iterations to reach objective point: 56 iterations -> 25 steps | Helped |
| start the editing stage at different initial timesteps (ablation) | DragNoise · point-based image editing | scale of controlled semantics | Mixed |
| optimize the bottleneck feature rather than encoder or decoder blocks (ablation) | DragNoise · point-based image editing | image quality and controllability | Helped |
| optimize encoder features instead of the bottleneck feature (ablation) | DragNoise · point-based image editing | generation quality | Hurt |
| optimize decoder features instead of the bottleneck feature (ablation) | DragNoise · point-based image editing | semantic editing direction | Hurt |
| halt propagation at t′=20 rather than t′=10 (ablation) | DragNoise · point-based image editing | generation clarity and detail preservation | Hurt |
| continue propagation through all denoising steps rather than halting at t′=10 (ablation) | DragNoise · point-based image editing | detail preservation | Hurt |
Surprising findings
- Optimizing only at one timestep without propagation leads to a forgetting issue where later denoising steps ignore the manipulation effect.
Limitations
- Handling real images while preserving fidelity relies on LoRA fine-tuning, which can limit editing range.
- Fails on intended rotations and tasks needing global perspective such as novel view synthesis.
- Edits may negate each other in some cases.
Experimental setup
Datasets: DragBench
Benchmarks: DragBench
Metrics: mean distance (MD) image fidelity (IF) MSE LPIPS optimization time
Baselines: DragGAN FreeDrag DragDiffusion
Builds on: DragGAN DragDiffusion DDIM inversion LoRA
Closest papers by meaning
- DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing (CVPR 2024 · similarity 0.91)
- Exploring Low-Dimensional Subspace in Diffusion Models for Controllable Image Editing (NeurIPS 2024 · similarity 0.89)
- The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing (ICLR 2024 · similarity 0.89)
- FastDrag: Manipulate Anything in One Step (NeurIPS 2024 · similarity 0.88)
- Localize, Understand, Collaborate: Semantic-Aware Dragging via Intention Reasoner (NeurIPS 2024 · similarity 0.88)
- Inversion-Free Image Editing with Language-Guided Diffusion Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.