Research map / Image generation / Editing and style transfer
Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes motion guidance, a zero-shot method that steers Stable Diffusion sampling with gradients through an off-the-shelf RAFT optical flow network plus a color-consistency loss so that edited images follow a user-specified dense flow field.
Problem
Precisely editing object layout, position, pose, and shape in images with diffusion models is difficult; existing motion-editing methods are limited to sparse motion inputs, require text, per-image finetuning, or specific architectures.
Approach
Use an off-the-shelf differentiable optical flow estimator as classifier-style guidance on one-step clean-image predictions, combined with a warp-based color loss, occlusion masking, edit masks, recursive denoising and gradient clipping, so dense flow fields can be imposed without training.
Key results
- Motion guidance achieves an attractive trade-off on Flow Loss vs. CLIP similarity against RePaint and SDEdit forward-warp baselines, which get low flow loss but low CLIP similarity.
- Qualitatively, InstructPix2Pix rarely moves objects significantly, and DragGAN produces many artifacts on out-of-domain real images.
- Ablations show removing recursive denoising, color loss, or occlusion masking causes artifacts, color changes, or ghosting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove recursive denoising (ablation) | Motion Guidance · Qualitative image edits | edit quality | Hurt |
| remove the color loss (ablation) | Motion Guidance · Qualitative image edits | appearance fidelity | Hurt |
| remove the flow loss and replace computed flow with target flow in the color loss (ablation) | Motion Guidance · Qualitative image edits | disocclusion quality | Hurt |
| remove occlusion masking (ablation) | Motion Guidance · Qualitative image edits | occlusion quality | Hurt |
| turn guidance off for the final 100 steps | Motion Guidance · Qualitative image edits | image detail quality | Helped |
| use fewer recursive denoising steps | Motion Guidance · Motion transfer | edit quality | Helped |
| use RePaint instead of Motion Guidance | Motion Guidance · Curated dataset and KITTI | Flow Loss | Mixed |
| use RePaint instead of Motion Guidance | Motion Guidance · Curated dataset and KITTI | CLIP Similarity | Hurt |
| use SDEdit with forward warping instead of Motion Guidance | Motion Guidance · Curated dataset and KITTI | CLIP Similarity | Hurt |
Surprising findings
- Optical flow networks are invariant to color changes, so flow loss alone causes objects to change color; a color loss is needed.
- Horizontal or vertical flips could not be achieved because such flows are out-of-distribution for the flow network.
Limitations
- Slow sampling: 15 to 70 minutes per sample depending on recursive denoising steps.
- Guidance is sometimes unstable and yields poorly converged samples; some flows (e.g., flips) fail.
- Forgetting or altering source content (identity, laptop screen), especially for real images without captions.
Experimental setup
Datasets: KITTI Unsplash images (curated) Stable Diffusion generated images (curated)
Benchmarks: curated motion-editing dataset KITTI-derived dataset
Metrics: Flow Loss (RAFT) Flow Loss (GMFlow) CLIP Similarity
Baselines: InstructPix2Pix SDEdit forward warp RePaint forward warp DragGAN GMFlow (flow metric)
Builds on: classifier guidance reconstruction guidance DDIM RePaint SDEdit
Closest papers by meaning
- DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing (CVPR 2024 · similarity 0.88)
- A Diffusion-Based Framework for Occluded Object Movement (AAAI 2025 · similarity 0.88)
- Motion-Zero: A Zero-Shot Trajectory Control Framework of Moving Object for Diffusion-Based Video Generation (AAAI 2025 · similarity 0.88)
- DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models (ICLR 2024 · similarity 0.88)
- MotionEditor: Editing Video Motion via Content-Aware Diffusion (CVPR 2024 · similarity 0.88)
- Spatiotemporal Skip Guidance for Enhanced Video Diffusion Sampling (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.