Research map / Image generation / Editing and style transfer
Learning Action and Reasoning-Centric Image Editing from Videos and Simulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Curates AROA dataset of truly minimal action- and reasoning-centric image edit pairs from videos and Kubric simulation, trains an InstructPix2Pix-based editor on it, and introduces the AROA-Bench benchmark plus a discriminative DiscEdit metric.
Problem
Instruction-guided image editing models perform poorly on action- and reasoning-centric edits, largely because available training pairs are noisy and not truly minimal; existing similarity metrics also reward copying the source image.
Approach
Build training triplets (source, prompt, target) where each pair contains a single meaningful change, sourced from carefully filtered video frame pairs (Action-Genome, Something-Something) and a Kubric simulator, and evaluate with human judgments plus a discriminative no-change vs. change metric (DiscEdit) instead of visual similarity.
Key results
- On AROA-Bench human judgment, the AURORA-finetuned model reaches an overall score of 41.3 versus 32.6 for MagicBrush and 10.8 for GenHowTo.
- On action (Something-Something) and reasoning (WhatsUp) human scores, the model reaches 31.8 and 27.3 versus 17.0 and 3.0 for MagicBrush.
- A naive copy of the source image outperforms MagicBrush on L1, L2, DINO and CLIP-I similarity metrics on the MagicBrush test set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train InstructPix2Pix on AROA instead of using MagicBrush | InstructPix2Pix image editing model · Action-Genome | human edit success: 16.3 -> 35.6 | Helped |
| Train InstructPix2Pix on AROA instead of using MagicBrush | InstructPix2Pix image editing model · Something-Something | human edit success: 17.0 -> 31.8 | Helped |
| Train InstructPix2Pix on AROA instead of using MagicBrush | InstructPix2Pix image editing model · WhatsUp | human edit success: 3.0 -> 27.3 | Helped |
| Train InstructPix2Pix on AROA instead of using MagicBrush | InstructPix2Pix image editing model · Kubric | human edit success: 9.3 -> 59.6 | Helped |
| Train InstructPix2Pix on AROA instead of using MagicBrush | InstructPix2Pix image editing model · CLEVR | human edit success: 22.0 -> 46.1 | Helped |
| Train InstructPix2Pix on AROA instead of using MagicBrush | InstructPix2Pix image editing model · MagicBrush | human edit success: 61.7 -> 60.5 | No clear effect |
| Train InstructPix2Pix on AROA instead of using MagicBrush | InstructPix2Pix image editing model · Epic Kitchen | human edit success: 17.7 -> 14.2 | Hurt |
| Train InstructPix2Pix on AROA instead of using MagicBrush | InstructPix2Pix image editing model · Emu-Global | human edit success: 42.3 -> 33.0 | Hurt |
| Train InstructPix2Pix on AROA instead of using MagicBrush | InstructPix2Pix image editing model · AROA-Bench | overall human edit success: 32.6 -> 41.3 | Helped |
| Train on AROA rather than MagicBrush | InstructPix2Pix image editing model · WhatsUp | DiscEdit accuracy: 0.472 -> 0.565 | Helped |
| Train on AROA rather than MagicBrush | InstructPix2Pix image editing model · Something-Something | DiscEdit accuracy: 0.371 -> 0.548 | Helped |
| Train on AROA rather than MagicBrush | InstructPix2Pix image editing model · Action-Genome | DiscEdit accuracy: 0.477 -> 0.583 | Helped |
Surprising findings
- Naively copying the source image scores better than the MagicBrush model on standard visual similarity metrics, showing these metrics reward faithfulness rather than editing.
- The MagicBrush model scores below random chance (0.472 and 0.371) on DiscEdit for some tasks.
Limitations
- Model still struggles with truly moving objects, often keeping the original and adding a new one, and swapping object positions fails.
- Action edits on complex real-world images remain challenging, with low human scores on Epic-Kitchen (14.2).
- DiscEdit performance is only slightly above random on most tasks, and the cause of MagicBrush's below-chance results was not identified.
Experimental setup
Datasets: AURORA MagicBrush Action-Genome-Edit Something-Something-Edit Kubric-Edit InstructPix2Pix GenHowTo HQ-Edit
Benchmarks: AROA-Bench MagicBrush test
Metrics: human edit success score (0/50/100) DiscEdit binary accuracy L1 L2 DINO CLIP-I
Baselines: InstructPix2Pix GenHowTo MGIE MagicBrush Naive Copy
Builds on: InstructPix2Pix MagicBrush Kubric Action Genome Something-Something v2
Closest papers by meaning
- Reasoning to Edit: Hypothetical Instruction-Based Image Editing with Visual Reasoning (ICML 2026 · similarity 0.90)
- UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits (CVPR 2026 · similarity 0.90)
- ReasonEdit: Towards Reasoning-Enhanced Image Editing Models (CVPR 2026 · similarity 0.89)
- EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing (ICLR 2026 · similarity 0.89)
- Meta-CoT: Enhancing Granularity and Generalization in Image Editing (CVPR 2026 · similarity 0.89)
- KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.