Research map / Image generation / Editing and style transfer
UltraEdit: Instruction-based Fine-Grained Image Editing at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds UltraEdit, a ~4M-sample instruction-based image editing dataset generated from real-image anchors, LLM-expanded instructions, and automatic region masks, and trains SD1.5 editors on it to set records on MagicBrush and Emu Edit.
Problem
Existing instruction-based image editing datasets have limited instruction diversity, biases inherited from text-to-image generators, and lack region-based editing data, limiting the performance of trained editors.
Approach
Combine LLM instruction expansion seeded by human-written examples, real-image anchors (SDEdit-style img2img from COCO/NoCaps etc. captions) with prompt-to-prompt target generation using SDXL-Turbo, and an automatic region-mask pipeline (RAM, LLM, GroundingDINO, SAM) with a modified inpainting schedule to create region-based edit samples.
Key results
- On MagicBrush single-turn, the model evaluated with region input reaches L1 0.0575 and CLIP-I 0.9307, versus 0.1141 L1 for IP2P zero-shot.
- On Emu Edit Test, the 3M free-form model reaches the best CLIPdir (0.1076) and beats Emu Edit (10M proprietary data) on CLIPdir and CLIPout trends at scale.
- Ablation: real image anchors give better Emu Edit metrics at 450K/1M/1.5M, and scaling only helps with anchors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on UltraEdit free-form data instead of InstructPix2Pix data | Stable Diffusion v1.5 editing model · MagicBrush test, single-turn | L1: 0.1141 -> 0.0689 | Helped |
| Add region-based editing data during training (ablation) | Stable Diffusion v1.5 editing model · MagicBrush test, single-turn | L1: 0.0689 -> 0.0614 | Helped |
| Add region input at evaluation (ablation) | UltraEdit-trained model · MagicBrush test, single-turn | L1: 0.0614 -> 0.0575 | Helped |
| Add region input at evaluation (ablation) | UltraEdit-trained model · MagicBrush test, multi-turn | DINO: 0.8322 -> 0.8505 | Helped |
| Add real image anchors during data generation (ablation) | UltraEdit-trained Stable Diffusion v1.5 · Emu Edit Test, 450k training samples | CLIPdir: 0.0728 -> 0.0823 | Helped |
| Add real image anchors during data generation (ablation) | UltraEdit-trained Stable Diffusion v1.5 · Emu Edit Test, 1M training samples | CLIPdir: 0.0638 -> 0.0925 | Helped |
| Add real image anchors during data generation (ablation) | UltraEdit-trained Stable Diffusion v1.5 · Emu Edit Test, 1.5M training samples | CLIPdir: 0.0720 -> 0.0952 | Helped |
| Increase UltraEdit free-form training data from 450k to 1M (ablation) | Stable Diffusion v1.5 editing model · Emu Edit Test | CLIPdir: 0.0823 -> 0.0862 | Helped |
| Increase UltraEdit free-form training data from 1M to 1.5M (ablation) | Stable Diffusion v1.5 editing model · Emu Edit Test | L1: 0.0515 -> 0.0600 | Hurt |
| Add region-based editing data to the training mixture (ablation) | UltraEdit-trained editing model · Emu Edit Test, free-form editing | CLIPout | Helped |
| Use region-based editing data for region-based editing tasks (ablation) | UltraEdit-trained editing model · Emu Edit Test, region-based editing | CLIPout | Mixed |
| Use region-based rather than free-form data | UltraEdit dataset samples · UltraEdit dataset quality assessment | SSIM: 0.6401 -> 0.7413 | Helped |
Surprising findings
- MagicBrush-trained IP2P scores high on MagicBrush but performs poorly on Emu Edit, indicating benchmark overfitting.
- Without real image anchors, scaling data does not yield consistent improvement, unlike with anchors.
Limitations
- Region-based data is small (~108K instances) relative to free-form data.
- Emu Edit Test has known caption quality issues that the authors had to address.
- Evaluation relies on automatic metrics (CLIP, DINO, L1) without human evaluation.
Experimental setup
Datasets: MS COCO NoCaps Flickr30k VizWiz Caption TextCaps Localized Narratives ShareGPT4V LAION-LVIS
Benchmarks: MagicBrush Emu Edit Test
Metrics: L1 L2 CLIP-I DINO CLIPdir CLIPout CLIPimg
Baselines: InstructPix2Pix HIVE MagicBrush Emu Edit Null Text Inversion SD-SDEdit GLIDE Blended Diffusion
Builds on: InstructPix2Pix prompt-to-prompt SDEdit SDXL-Turbo Emu Edit
Closest papers by meaning
- AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea (CVPR 2025 · similarity 0.92)
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.91)
- ImgEdit: A Unified Image Editing Dataset and Benchmark (NeurIPS 2025 · similarity 0.91)
- FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Model (CVPR 2025 · similarity 0.90)
- Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing (ICML 2025 · similarity 0.89)
- Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.