Research map / Vision-language and multimodal / Multimodal LLMs
SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Supplementary material for SmartEdit, an MLLM-based instruction image editor, detailing training, a synthetic complex-instruction data pipeline, Reason-Edit evaluation with human Ins-align, and MagicBrush comparisons.
Problem
Instruction-based image editing methods built on CLIP text encoders fail on complex understanding and reasoning instructions.
Approach
Couple an MLLM (LLaVA) with a diffusion editor via a QFormer aligned to the CLIP text encoder and a Bidirectional Interaction Module (BIM), plus a synthetic dataset of ~476 pairs for complex understanding and reasoning scenarios.
Key results
- On Reason-Edit reasoning scenarios, SmartEdit-13B reaches Ins-align 0.817 vs 0.483 for InstructDiffusion (human evaluation).
- On the MagicBrush test set, SmartEdit-13B obtains CLIP-T 0.305, above MagicBrush-168 (0.302), while MagicBrush-168 leads on L1, CLIP-I and DINO-I.
- In a 25-person user study, over 67% judged SmartEdit better aligned with instructions and over 72% preferred its image quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SmartEdit-13B instead of SmartEdit-7B | SmartEdit · Reason-Edit understanding scenarios | Ins-align: 0.712 -> 0.771 | Helped |
| Use SmartEdit-13B instead of SmartEdit-7B | SmartEdit · Reason-Edit reasoning scenarios | Ins-align: 0.789 -> 0.817 | Helped |
| Use SmartEdit rather than MagicBrush-168 | image editing methods · MagicBrush test set | CLIP-T: 0.302 -> 0.303; 0.302 -> 0.305 | Helped |
| Add BIM instead of using Plain (ablation) | SmartEdit-7B · Reason-Edit understanding scenarios | Ins-align: 0.695 -> 0.712 | Helped |
| Add BIM instead of using SimpleCA (ablation) | SmartEdit-7B · Reason-Edit understanding scenarios | Ins-align: 0.692 -> 0.712 | Helped |
| Add BIM instead of using Plain (ablation) | SmartEdit-7B · Reason-Edit reasoning scenarios | Ins-align: 0.694 -> 0.789 | Helped |
| Add BIM instead of using SimpleCA (ablation) | SmartEdit-7B · Reason-Edit reasoning scenarios | Ins-align: 0.722 -> 0.789 | Helped |
| Add segmentation data to editing data (ablation) | SmartEdit-7B · Reason-Edit understanding scenarios | Ins-align: 0.201 -> 0.361 | Helped |
| Add synthetic editing data to editing data (ablation) | SmartEdit-7B · Reason-Edit understanding scenarios | Ins-align: 0.201 -> 0.440 | Helped |
| Add segmentation data to editing data (ablation) | SmartEdit-7B · Reason-Edit reasoning scenarios | Ins-align: 0.233 -> 0.311 | Helped |
| Add synthetic editing data to editing data (ablation) | SmartEdit-7B · Reason-Edit reasoning scenarios | Ins-align: 0.233 -> 0.567 | Helped |
Surprising findings
- Standard metrics (PSNR/SSIM/LPIPS/CLIP Score, L1/CLIP-I/DINO-I) often disagree with visual judgment, with MagicBrush-168 scoring best on several metrics despite visibly poorer edits.
- Ablation shows the synthetic editing dataset yields large gains in Ins-align, while the plain (no BIM) variant underperforms in reasoning.
Limitations
- Ins-align relies on manual evaluation by four workers, which is costly and not automated.
- Standard automatic metrics are shown to be unreliable for instruction-based editing evaluation.
- Synthetic editing data is small (about 476 pairs) and partly manually filtered.
Experimental setup
Datasets: CC12M COCOStuff RefCOCO GRefCOCO LISA reasoning segmentation InstructPix2Pix MagicBrush LLaVA-Instruct-150k
Benchmarks: Reason-Edit MagicBrush test set
Metrics: PSNR SSIM LPIPS CLIP Score Ins-align L1 CLIP-I CLIP-T DINO-I user preference
Baselines: InstructPix2Pix MagicBrush InstructDiffusion MGIE MagicBrush-52 MagicBrush-168
Builds on: InstructPix2Pix InstructDiffusion LLaVA MGIE
Closest papers by meaning
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.94)
- ReasonEdit: Towards Reasoning-Enhanced Image Editing Models (CVPR 2026 · similarity 0.92)
- Guiding Instruction-based Image Editing via Multimodal Large Language Models (ICLR 2024 · similarity 0.92)
- Reasoning to Edit: Hypothetical Instruction-Based Image Editing with Visual Reasoning (ICML 2026 · similarity 0.91)
- AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea (CVPR 2025 · similarity 0.91)
- FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Model (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.