Research map / Vision-language and multimodal / Multimodal LLMs
InsightEdit: Towards Better Instruction Following for Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InsightEdit, an MLLM-conditioned instruction-based image editor with a two-stream (text Q-Former and image adapter) bridging mechanism, trained on a new automatically constructed AdvancedEdit dataset of 2.5M mask-based editing pairs.
Problem
Instruction-based image editing struggles with complex instruction following and background consistency, and existing datasets have low resolution, weak background consistency, and simplistic template instructions.
Approach
Builds an automated data pipeline (MLLM captioning, GroundedSAM masks, mask-based inpainting for target images, GPT-4o-recaptioned complex instructions, VIEScore filtering) and conditions a diffusion editor on both MLLM-reasoned text features (via Q-Former and BIM) and image features aligned to the target image's CLIP embedding (IAA), injected through decoupled cross-attention.
Key results
- On AdvancedEdit-Eval, InsightEdit with AdvancedEdit reaches VIEScore 0.831 vs 0.682 for SmartEdit-7B, with PSNR 22.871 and LPIPS 0.071.
- On Reason-Edit (understanding), InsightEdit with AdvancedEdit reaches VIEScore 0.934 vs 0.866 for SmartEdit-7B.
- Ablation: adding IAA raises VIEScore from 7.307 to 7.545 and lowers LPIPS from 0.095 to 0.071; AdvancedEdit data raises VIEScore from 6.974 (none) to 7.545.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train InsightEdit with AdvancedEdit instead of without AdvancedEdit | InsightEdit · AdvancedEdit-Eval | VIEScore: 0.738 -> 0.831 | Helped |
| Train InsightEdit with AdvancedEdit instead of without AdvancedEdit | InsightEdit · Reason-Edit understanding scenarios | VIEScore: 0.901 -> 0.934 | Helped |
| Train InsightEdit with AdvancedEdit instead of without AdvancedEdit | InsightEdit · Reason-Edit reasoning scenarios | VIEScore: 0.893 -> 0.947 | Helped |
| Add the IAA module (ablation) | InsightEdit · AdvancedEdit-Eval | PSNR: 22.348 -> 22.871 | Helped |
| Add the IAA module (ablation) | InsightEdit · AdvancedEdit-Eval | VIEScore: 7.307 -> 7.545 | Helped |
| Train on SimpleEdit instead of no added editing data (ablation) | InsightEdit · AdvancedEdit-Eval | PSNR: 21.267 -> 22.260 | Helped |
| Use AdvancedEdit instead of SimpleEdit (ablation) | InsightEdit · AdvancedEdit-Eval | CLIPScore: 20.557 -> 21.002 | Helped |
| Use AdvancedEdit instead of SimpleEdit (ablation) | InsightEdit · AdvancedEdit-Eval | VIEScore: 7.213 -> 7.545 | Helped |
Surprising findings
- On Reason-Edit reasoning scenarios, InsightEdit slightly trails SmartEdit in CLIPScore (20.867 vs 20.950).
Limitations
- Only 202,822 of the 2.5M AdvancedEdit pairs were used due to limited resources.
- Relies on a single MLLM (LLaVA-7B) and an older diffusion backbone; authors suggest stronger MLLMs and diffusion models could help.
- Evaluation relies heavily on VIEScore (MLLM-based) and PSNR/SSIM against a single reference, which may not fully reflect human preference.
Experimental setup
Datasets: CC12M InstructPix2Pix MagicBrush COCO RefCOCO GRefCOCO COCOStuff LISA ReasonEdit Pexels
Benchmarks: AdvancedEdit-Eval Reason-Edit
Metrics: VIEScore CLIPScore PSNR SSIM LPIPS
Baselines: InstructPix2Pix MagicBrush InstructDiffusion MGIE SmartEdit
Builds on: InstructPix2Pix SmartEdit GILL IP-Adapter VIEScore GroundedSAM
Closest papers by meaning
- SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models (CVPR 2024 · similarity 0.94)
- AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea (CVPR 2025 · similarity 0.93)
- FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Model (CVPR 2025 · similarity 0.91)
- Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing (ICML 2025 · similarity 0.91)
- UltraEdit: Instruction-based Fine-Grained Image Editing at Scale (NeurIPS 2024 · similarity 0.91)
- Guiding Instruction-based Image Editing via Multimodal Large Language Models (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.