Research map / Image generation / Editing and style transfer
Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ICEdit, an in-context instruction image-editing method built on FLUX.1 Fill that uses a diptych prompt, a light LoRA-MoE fine-tune on 50K samples, and VLM-based early-filter noise selection at inference.
Problem
Instruction-based image editing faces a precision-efficiency tradeoff: fine-tuned methods need massive datasets (often >10M pairs) and compute, while training-free methods follow instructions poorly.
Approach
Reframe editing as in-context generation on a diptych (source left, edit right) using an inpainting DiT with a fixed mask and an in-context instruction prompt, then apply minimal LoRA-MoE fine-tuning and an Early Filter inference-time scaling that uses a VLM to pick the best initial noise from early denoising steps.
Key results
- On the MagicBrush test set, ICEdit reaches L1 0.060, CLIP-I 0.928 and DINO 0.853, the best among compared methods.
- With inference-time scaling, ICEdit's VIE overall score of 78.2 surpasses SeedEdit's 75.7.
- Using 50K fine-tuning samples (about 0.05M) versus 10M for Emu Edit, ICEdit reaches GPT score 0.68 on the Emu test set, close to Emu Edit's 0.72.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the IC prompt to training-free editing (ablation) | FLUX.1 Fill in-context editing · Emu Test set | GPT: 0.14 -> 0.24 | Helped |
| add the IC prompt to fine-tuned ICEdit (ablation) | ICEdit · Emu Test set | GPT: 0.62 -> 0.68 | Helped |
| replace standard LoRA rank 64 with the full LoRA-MoE design (ablation) | ICEdit · Emu Test set | GPT: 0.60 -> 0.68 | Helped |
| fine-tune all modules rather than only the MoE module (ablation) | ICEdit · Emu Test set | GPT: 0.51 -> 0.68 | Helped |
| add VLM inference-time noise filtering with 6 noise samples and 10 early steps (ablation) | ICEdit · Emu Test set | GPT: 0.68 -> 0.78 | Helped |
| use 10 rather than 4 early steps for VLM filtering (ablation) | ICEdit inference-time scaling · Emu Test set | GPT: 0.72 -> 0.78 | Helped |
| increase VLM-filtered noise samples from 6 to 12 (ablation) | ICEdit inference-time scaling · Emu Test set | GPT: 0.78 -> 0.79 | Helped |
| replace VLM filtering with CLIP filtering (ablation) | ICEdit inference-time scaling · Emu Test set | GPT: 0.80 -> 0.65 | Hurt |
| increase expert count from 4 to 6 (ablation) | ICEdit MoE · Emu Test set | GPT: 0.68 -> 0.66 | Hurt |
| increase expert count from 4 to 8 (ablation) | ICEdit MoE · Emu Test set | GPT: 0.68 -> 0.61 | Hurt |
Surprising findings
- Smaller VLMs (7B) often misjudge edit quality, so a 72B judge is needed for inference-time filtering.
- CLIP-based selection of noise seeds degraded GPT-based scores relative to no selection (0.65 vs 0.68).
Limitations
- Object movement instructions fail due to limited motion-oriented training data.
- Polysemous terms are misresolved by the T5 text encoder (e.g., mouse as device vs animal).
- Inference-time scaling depends on a 72B VLM judge; smaller 7B judges misjudge quality.
Experimental setup
Datasets: MagicBrush OmniEdit Emu Edit test set MagicBrush test set
Benchmarks: Emu Edit test set MagicBrush test set
Metrics: CLIP-I CLIP-Out DINO GPT-4o score L1 VIE-Score SC PQ
Baselines: InstructPix2Pix MagicBrush Emu Edit UltraEdit FluxEdit FLUX.1 Fill RF-Solver-Edit ACE++ SeedEdit
Builds on: FLUX.1 Fill LoRA Mixture of Experts VIE-Score OmniEdit
Closest papers by meaning
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.90)
- AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea (CVPR 2025 · similarity 0.90)
- SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models (CVPR 2024 · similarity 0.90)
- Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks (AAAI 2024 · similarity 0.89)
- FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Model (CVPR 2025 · similarity 0.89)
- UltraEdit: Instruction-based Fine-Grained Image Editing at Scale (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.