Research map / Video / Video generation
Instruction-based Image Manipulation by Watching How Things Move
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 6M-pair instruction-editing dataset from real video frame pairs with MLLM-generated instructions, and fine-tunes Stable Diffusion 1.5 with a spatial conditioning strategy (source image concatenated along width) for non-rigid and viewpoint image edits.
Problem
Instruction-based image editing models struggle with non-rigid edits (pose, expression, viewpoint) because training datasets rely on synthetically generated target images that have artifacts and identity drift.
Approach
Sample frame pairs from internet videos with optical-flow-based motion filtering, use MLLMs (GPT-4o, Pixtral-12B) to write editing instructions from the real frame pair, and condition a diffusion model on the source image by width-wise spatial concatenation instead of channel concatenation.
Key results
- Ours achieves CLIP-D 0.1361 and CLIP-Inst 0.8724 on the authors' benchmark, the highest on both instruction-alignment metrics versus six baselines.
- In a 40-participant human study, users preferred Ours in 87.62% of cases over Imagic, InstructPix2Pix, and MagicBrush.
- Training on the InstructPix2Pix dataset with spatial conditioning gives CLIP-D 0.1277 vs 0.1361 for the authors' data, and channel conditioning gives CLIP-I 0.8552 vs 0.9275 for spatial conditioning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use video-frame training data instead of InstructPix2Pix data (ablation) | InstructMove · Non-rigid editing benchmark | CLIP-D: 0.1277 -> 0.1361 | Helped |
| Use video-frame training data instead of InstructPix2Pix data (ablation) | InstructMove · Non-rigid editing benchmark | CLIP-Inst: 0.8414 -> 0.8724 | Helped |
| Use video-frame training data instead of InstructPix2Pix data (ablation) | InstructMove · Non-rigid editing benchmark | CLIP-I: 0.9094 -> 0.9275 | Helped |
| Use spatial conditioning instead of channel conditioning (ablation) | InstructMove · Non-rigid editing benchmark | CLIP-D: 0.0853 -> 0.1361 | Helped |
| Use spatial conditioning instead of channel conditioning (ablation) | InstructMove · Non-rigid editing benchmark | CLIP-Inst: 0.8679 -> 0.8724 | Helped |
| Use spatial conditioning instead of channel conditioning (ablation) | InstructMove · Non-rigid editing benchmark | CLIP-I: 0.8552 -> 0.9275 | Helped |
| Train InstructMove rather than MagicBrush | Non-rigid editing methods · Non-rigid editing benchmark | CLIP-D: 0.0972 -> 0.1361 | Helped |
| Train InstructMove rather than MagicBrush | Non-rigid editing methods · Non-rigid editing benchmark | CLIP-Inst: 0.8648 -> 0.8724 | Helped |
| Train InstructMove rather than InstructPix2Pix | Non-rigid editing methods · Non-rigid editing benchmark | CLIP-I: 0.9380 -> 0.9275 | Hurt |
| Use InstructMove rather than the compared editing methods | Image editing methods · Human preference study | Preference: 5.0% -> 87.62% | Helped |
Surprising findings
- Baselines with higher content-preservation scores (CLIP-I) often left inputs nearly unchanged, so CLIP-I alone is misleading for edit quality.
Limitations
- MLLMs sometimes produce inaccurate instructions or miss transformations, causing unintended changes such as slight viewpoint shifts.
- Cannot isolate specific objects accurately from instructions.
- Trained only on realistic transformations, so it cannot handle style transfer or object replacement without merging other datasets.
Experimental setup
Datasets: 6M video frame pairs with MLLM-generated instructions InstructPix2Pix dataset (ablation)
Benchmarks: Authors' 50-image non-rigid editing benchmark
Metrics: CLIP-D CLIP-Inst CLIP-I human preference
Baselines: NullTextInversion MasaCtrl Imagic InstructPix2Pix MagicBrush UltraEdit
Builds on: InstructPix2Pix Stable Diffusion ControlNet T2I-Adapter RAFT
Closest papers by meaning
- Guiding Instruction-based Image Editing via Multimodal Large Language Models (ICLR 2024 · similarity 0.89)
- Learning Action and Reasoning-Centric Image Editing from Videos and Simulation (NeurIPS 2024 · similarity 0.87)
- Dynamic Motion Blending for Versatile Motion Editing (CVPR 2025 · similarity 0.87)
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.87)
- Señorita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists (NeurIPS 2025 · similarity 0.87)
- Text-Vision Co-Instructed Image Editing (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.