Research map / Image generation / Editing and style transfer
Emu Edit: Precise Image Editing via Recognition and Generation Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Emu Edit trains a diffusion-based instruction image editor multi-task across 16 editing and vision tasks with learned task embeddings, enabling few-shot adaptation via task inversion, and releases a seven-category editing benchmark.
Problem
Instruction-based image editing models often fail to accurately follow user instructions and generalize poorly to tasks slightly outside their training distribution.
Approach
Train a single instruction-conditioned diffusion editor jointly on 16 editing and computer-vision tasks (formulated as generation), condition it on learned per-task embeddings predicted from the instruction by a Flan-T5-XL classifier, and adapt to new tasks by learning only a new task embedding (task inversion).
Key results
- On the MagicBrush test set, Emu Edit reaches CLIPdir 0.135 and DINO 0.879 versus 0.123 and 0.871 for MagicBrush, and 0.115 and 0.767 for InstructPix2Pix.
- Human raters preferred Emu Edit over baselines by a large margin on text alignment and image faithfulness (e.g., 77% vs InstructPix2Pix on the Emu Edit test set ratings reported as baseline-preference complements).
- Task inversion with a single or 100 examples nearly matches a 100k-sample expert on new tasks such as super-resolution, contour detection, and inpainting, performing similarly to full fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train Emu Edit across diverse editing and vision tasks instead of using the strongest instruction-editing baseline | Emu Edit · Emu Edit test set | CLIPdir: 0.090 -> 0.109 | Helped |
| Train Emu Edit across diverse editing and vision tasks instead of using the strongest instruction-editing baseline | Emu Edit · MagicBrush test set | CLIPdir: 0.123 -> 0.135 | Helped |
| Train Emu Edit across diverse editing and vision tasks instead of using the strongest instruction-editing baseline | Emu Edit · MagicBrush test set | L1: 0.058 -> 0.052 | Helped |
| Add learned task embeddings (ablation) | Emu Edit · Emu Edit benchmark validation set | CLIPdir: 0.104 -> 0.119 | Helped |
| Use predicted task type rather than ground-truth task type (ablation) | Emu Edit · Emu Edit benchmark validation set | CLIPdir: 0.119 -> 0.117 | Hurt |
| Add detection and segmentation tasks (ablation) | Emu Edit · Region-based editing | Text alignment | Helped |
| Add image-to-image translation tasks (ablation) | Emu Edit · Free-form editing | Text alignment | Helped |
| Increase the number of training tasks (ablation) | Emu Edit · Style and Texture tasks | CLIPdir | Helped |
| Train one model on multiple tasks rather than separate expert models (ablation) | Emu Edit · Individual image-editing tasks | performance | Helped |
Surprising findings
- Adding computer vision tasks such as detection and segmentation improves editing performance on region-based edits, as validated by human raters and metrics.
- Multi-task training on all tasks outperforms training expert models on individual tasks.
Limitations
- Baselines and Emu Edit struggle with counting and complex relational instructions, as noted in the conclusion (needs multimodal LLM integration).
- Text-based baselines (PnP, Null-Text Inversion) were given ground-truth captions, which may bias comparisons.
- Training data is synthetic and filtered heavily (70% removed), with the MagicBrush and Emu Edit benchmarks sharing input images from MagicBrush.
Experimental setup
Datasets: MagicBrush test set MS-COCO images (via MagicBrush) Emu Edit training set (10M synthetic)
Benchmarks: Emu Edit benchmark MagicBrush test set
Metrics: CLIPdir CLIPimg CLIPout L1 DINO similarity human text alignment preference human image faithfulness preference
Baselines: InstructPix2Pix MagicBrush PnP Null-Text Inversion
Builds on: Emu InstructPix2Pix Prompt-to-Prompt Latent diffusion model Flan-T5
Closest papers by meaning
- AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea (CVPR 2025 · similarity 0.88)
- ImgEdit: A Unified Image Editing Dataset and Benchmark (NeurIPS 2025 · similarity 0.87)
- UltraEdit: Instruction-based Fine-Grained Image Editing at Scale (NeurIPS 2024 · similarity 0.87)
- Meta-CoT: Enhancing Granularity and Generalization in Image Editing (CVPR 2026 · similarity 0.87)
- Guiding Instruction-based Image Editing via Multimodal Large Language Models (ICLR 2024 · similarity 0.87)
- Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.