Research map / Image generation / Editing and style transfer
Instruction-based diffusion editing
Edit images from text instructions or exemplars using trained diffusion models.
151 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Editing and style transfer, keyword cluster “image editing · edits · inversion”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
36 papers in 2023–24 and 103 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 13.2% to 26.2%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Language-Based Image Editing With Recurrent Attentive Models (CVPR 2018 · 150 citations)
- 2019 · Controllable Unsupervised Text Attribute Transfer via Editing Entangled Latent Representation (NeurIPS 2019 · 68 citations)
- 2020 · Image2StyleGAN++: How to Edit the Embedded Images? (CVPR 2020 · 518 citations)
- 2021 · Learning by Planning: Language-Guided Global Image Editing (CVPR 2021 · 31 citations)
- 2022 · Blended Diffusion for Text-Driven Editing of Natural Images (CVPR 2022 · 755 citations)
- 2023 · InstructPix2Pix: Learning To Follow Image Editing Instructions (CVPR 2023 · 1,395 citations)
- 2024 · Emu Edit: Precise Image Editing via Recognition and Generation Tasks (CVPR 2024 · 81 citations)
- 2025 · AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea (CVPR 2025 · 28 citations)
- 2026 · Insert Anything: Image Insertion via In-Context Editing in DiT (AAAI 2026 · 8 citations)
Earliest papers here
- Language-Based Image Editing With Recurrent Attentive Models (CVPR 2018 · 150 citations)
- Controllable Unsupervised Text Attribute Transfer via Editing Entangled Latent Representation (NeurIPS 2019 · 68 citations)
- Image2StyleGAN++: How to Edit the Embedded Images? (CVPR 2020 · 518 citations)
What has been tried
91 papers here report 939 measured changes (a component added, removed or swapped, compared with a baseline): helped 73%, no clear effect 4%, hurt 21%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add learned task embeddings | Emu Edit benchmark validation set · CLIPdir | 0.104 -> 0.119 | Helped | Emu Edit: Precise Image Editing via Recognition and Generation Tasks (CVPR 2024 · 81 citations) |
| Add cycle-consistency augmentation | Synthetic evaluation dataset and 1K evaluation dataset · human preference | Helped | HIVE: Harnessing Human Feedback for Instructional Visual Editing (CVPR 2024 · 50 citations) | |
| Add the FFT-based edge smoother when compositing the edited layer | Local image editing · edge artifacts | Helped | ZONE: Zero-Shot Instruction-Guided Local Editing (CVPR 2024 · 35 citations) | |
| Extract masks only at the first denoising step rather than at later steps | Image editing · Editing quality | Helped | Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulation (CVPR 2024 · 32 citations) | |
| Add task-aware routing | EMU-Edit Test · CLIPim | 0.838 -> 0.872 | Helped | AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea (CVPR 2025 · 28 citations) |
| Omit the 2nd level of multi-level guidance | Tensor4D dataset, dynamic scene · FID | 18.59 -> 45.19 | Hurt | Control4D: Efficient 4D Portrait Editing with Text (CVPR 2024 · 25 citations) |
| Add Region-Embedder to fine-tuned InstructPix2Pix | RefCOCO-Edit · AP | 4.56 -> 4.64 | Helped | Referring Image Editing: Object-level Image Editing via Referring Expressions (CVPR 2024 · 24 citations) |
| add the IC prompt to training-free editing | Emu Test set · GPT | 0.14 -> 0.24 | Helped | Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer (NeurIPS 2025 · 16 citations) |
| add adversarial loss | Text-driven editing · FID | 162.275 -> 137.740 | Helped | GenN2N: Generative NeRF2NeRF Translation (CVPR 2024 · 15 citations) |
| Inject semantic guidance during the first 30–50 of 100 sampling steps instead of only the first 5–10 steps | CelebA-HQ 256×256, male attribute · edit effectiveness | Helped | Latent Space Editing in Transformer-Based Flow Matching (AAAI 2024 · 14 citations) | |
| add gated injection | Tamper-Syn2k · MSE | 0.0136 -> 0.0120 | Helped | Choose What You Need: Disentangled Representation Learning for Scene Text Recognition Removal and Editing (CVPR 2024 · 14 citations) |
| add pose-warped texture | WPose, 256x256 · FID | 27.571 -> 31.352 | Helped | UniHuman: A Unified Model For Editing Human Images in the Wild (CVPR 2024 · 11 citations) |
What reviewers will expect
Datasets and baselines named in the 52 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MagicBrush · 11 of 52 papers
- PIE-Bench · 7 of 52 papers
- InstructPix2Pix dataset · 4 of 52 papers
- AnyEdit · 2 of 52 papers
- COCO · 2 of 52 papers
- DragBench · 2 of 52 papers
- OmniEdit · 2 of 52 papers
Baselines
- InstructPix2Pix · 21 of 52 papers
- MagicBrush · 13 of 52 papers
- HIVE · 7 of 52 papers
- UltraEdit · 7 of 52 papers
- MasaCtrl · 6 of 52 papers
- SDEdit · 5 of 52 papers
- Null-text inversion · 5 of 52 papers
- Emu Edit · 4 of 52 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 139 papers; from OpenAlex and the papers' affiliations).
- Peking University · 8 papers
- Tsinghua University · 6 papers
- Zhejiang University · 5 papers
- National University of Singapore · 5 papers
- Shanghai Jiao Tong University · 4 papers
- Huawei · 4 papers
- Hong Kong University of Science and Technology · 4 papers
- Korea Advanced Institute of Science and Technology · 4 papers
- Meta · 4 papers
- University of Science and Technology of China · 3 papers
Use it
Download BibTeX of 33 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Inversion-based diffusion editing (113 papers)
- Localized and attribute-specific editing (73 papers)
- GAN inversion for editing (25 papers)
Elsewhere in the topic
- Texture and sketch synthesis (130 papers)
- Image-to-image translation (127 papers)
- Arbitrary neural style transfer (124 papers)
- Identity-preserving face manipulation (88 papers)
- Diffusion face generation and editing (61 papers)
- Virtual try-on (46 papers)
- Pose-guided person image synthesis (29 papers)
- Talking face generation (21 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).