Research map / Image generation / Editing and style transfer
Image Editing As Programs with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IEAP, a DiT-based instruction editing framework that decomposes edits into five atomic operations sequenced by a VLM/CoT planner and executed by LoRA-adapted FLUX.1-dev modules, outperforming baselines on MagicBrush and AnyEdit.
Problem
Instruction-driven image editing with diffusion transformers struggles on structurally-inconsistent edits requiring layout changes (add, remove, move, resize).
Approach
Reduce any editing instruction into a sequence of five atomic operations (RoI localization, RoI inpainting, RoI editing, RoI compositing, global transformation), planned by GPT-4o chain-of-thought and executed by LoRA adapters on a shared FLUX.1-dev DiT backbone, with LLM-guided layout updates for move/resize.
Key results
- On AnyEdit test, IEAP reaches CLIPim 0.882, L1 0.096, DINO 0.825 and GPT score 4.41, the best among compared methods (ICEdit GPT 4.13).
- On MagicBrush test, IEAP achieves CLIPim 0.922, CLIPout 0.247 and DINO 0.897, the best on these metrics, with a slightly higher L1 than ICEdit (0.060 vs 0.058).
- Per-category results show largest gains on local semantic editing (e.g., Add/Move/Resize), with IEAP's GPT average of 4.20 on Add vs 4.72 best baseline on Textual change being a weak spot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoT reasoning and instruction reduction into sequential atomic operations instead of end-to-end editing (ablation) | IEAP · AnyEdit local semantic editing test set | performance across all metrics | Helped |
| Use specialized RoI inpainting instead of a generic inpainting model (ablation) | IEAP · AnyEdit local semantic editing test set | performance | Helped |
| Use specialized RoI editing instead of a generic inpainting model (ablation) | IEAP · AnyEdit local semantic editing test set | performance | Helped |
| Use LLM-guided layout reconfiguration instead of random layout modifications (ablation) | IEAP · AnyEdit local semantic editing test set | performance | Helped |
| Use annular mask integration (ablation) | IEAP · AnyEdit local semantic editing test set | performance | Helped |
| Use IEAP instead of ICEdit | Image editing · MagicBrush test | L1: 0.058 -> 0.060 | Hurt |
| Use IEAP instead of ICEdit | Image editing · MagicBrush test | CLIP_im: 0.913 -> 0.922 | Helped |
| Use IEAP instead of ICEdit | Image editing · AnyEdit test | GPT: 4.13 -> 4.41 | Helped |
Surprising findings
- Baseline DiT editors score well on structure-preserving edits but degrade sharply on add/remove/move/resize, with some cases failing completely.
- On the AnyEdit Textual Change category, IEAP is not the top method on GPT average (4.20 vs ICEdit 4.43).
Limitations
- Complex shadow changes are sometimes inconsistent after compositing.
- Multiple editing iterations can cause progressive image quality decay.
Experimental setup
Datasets: AnyEdit CelebHQ-FM
Benchmarks: MagicBrush test AnyEdit test
Metrics: CLIP-image CLIP-out L1 DINO GPT-4o score
Baselines: InstructPix2Pix MagicBrush UltraEdit ICEdit SeedEdit (Doubao) Gemini 2.0 Flash GPT-4o
Builds on: FLUX.1-dev LoRA AnyEdit OminiControl Chain-of-Thought prompting
Closest papers by meaning
- Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer (NeurIPS 2025 · similarity 0.88)
- PAIR Diffusion: A Comprehensive Multimodal Object-Level Image Editor (CVPR 2024 · similarity 0.88)
- Object-Aware Inversion and Reassembly for Image Editing (ICLR 2024 · similarity 0.88)
- FunEditor: Achieving Complex Image Edits via Function Aggregation with Diffusion Models (AAAI 2025 · similarity 0.88)
- Inversion-Free Image Editing with Language-Guided Diffusion Models (CVPR 2024 · similarity 0.88)
- h-Edit: Effective and Flexible Diffusion-Based Editing via Doob's h-Transform (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.