Research map / Vision-language and multimodal / Multimodal LLMs
Guiding Instruction-based Image Editing via Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MGIE, which uses an MLLM (LLaVA-7B) to derive concise visually-grounded expressive instructions and passes learned visual tokens through an edit head into a Stable Diffusion editor, trained end-to-end on IPr2Pr.
Problem
Instruction-based image editing struggles when human instructions are brief or ambiguous, and CLIP text encoders capture static descriptions rather than the transformation needed for editing.
Approach
An MLLM perceives the image and the instruction to produce a concise expressive instruction, and appended learnable [IMG] tokens from its hidden states are mapped by an edit head into latent guidance for a diffusion editor; the MLLM and diffusion model are trained jointly end-to-end with instruction and editing losses.
Key results
- Zero-shot on MagicBrush: MGIE reaches 82.22 DINO, 91.14 CVS and 30.40 CTS, versus 71.46 DINO and 29.34 CTS for InstructPix2Pix.
- Zero-shot on MA5k: MGIE improves SSIM to 66.25 and LPIPS to 0.298 compared with InstructPix2Pix's 58.92 and 0.359.
- Expressive instructions average about 9.2 s per edit at inference, versus over 200 s for Null-Inv and Text2LIVE, and MGIE keeps inference cost close to InstructPix2Pix.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use visual-aware expressive instructions instead of instruction-only LLaMA guidance | MGIE vs. LGIE · EVR, GIER, MA5k, and MagicBrush (zero-shot) | editing metrics: MGIE: EVR 71.49 DINO vs. 69.71; EVR CVS 81.73 vs. 82.04; GIER 59.24 SSIM vs. 56.86; MA5k 66.25 SSIM vs. 64.60; MagicBrus | Mixed |
| Include instruction loss to train concise expressive instructions (ablation) | MGIE · MA5k | SSIM: 58.18 -> 66.25 | Helped |
| Include instruction loss to train concise expressive instructions (ablation) | MGIE · MA5k | LPIPS: 0.365 -> 0.298 | Helped |
| Include instruction loss to train concise expressive instructions (ablation) | MGIE · MagicBrush | DINO: 71.50 -> 82.22 | Helped |
| Include instruction loss to train concise expressive instructions (ablation) | MGIE · MagicBrush | CVS: 85.19 -> 91.14 | Helped |
| Include instruction loss to train concise expressive instructions (ablation) | MGIE · MagicBrush | CTS: 29.11 -> 30.40 | Helped |
| Use visual-aware MGIE guidance rather than LGIE guidance (ablation) | Frozen InsPix2Pix (FZ) · MA5k and MagicBrush | editing performance | Helped |
| Fine-tune InsPix2Pix on expressive instructions (FT) (ablation) | InsPix2Pix · MA5k and MagicBrush | editing performance | Helped |
| Jointly train the diffusion editor and language model end-to-end (E2E) (ablation) | MGIE · MA5k and MagicBrush | editing performance | Helped |
| Remove instruction loss (ablation) | LGIE and MGIE · MA5k and MagicBrush | editing metrics | Hurt |
| Use summarized expressive instructions instead of full descriptions (ablation) | MGIE · image-instruction alignment | CLIP-Score: 22.7 vs. 64.5 tokens | Helped |
| Use the What prompt instead of the How prompt (ablation) | MGIE · expressive instruction generation | visual alignment | Helped |
Surprising findings
- Plugging derived expressive instructions into a frozen InstructPix2Pix (FZ) can hurt performance, and LGIE (text-only LLM guidance) even underperforms the baseline in some ablations.
Limitations
- Compositional multi-step edits are hard to complete in one step.
- Weak language grounding and numerical perception, e.g., editing only one of several objects.
- Inherits biases from LLaVA and Stable Diffusion.
Experimental setup
Datasets: IPr2Pr EVR GIER MA5k MagicBrush
Benchmarks: EVR GIER MA5k MagicBrush
Metrics: L1 DINO similarity CLIP visual similarity SSIM LPIPS CLIP text similarity FID human ranking
Baselines: InstructPix2Pix LGIE Text2LIVE Null-Inv HIVE InstEdit
Builds on: InstructPix2Pix LLaVA GILL Stable Diffusion BLIP-2
Closest papers by meaning
- SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models (CVPR 2024 · similarity 0.92)
- RetouchIQ: MLLM Agents for Instruction-Based Image Retouching with Generalist Reward (CVPR 2026 · similarity 0.91)
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.90)
- ReasonEdit: Towards Reasoning-Enhanced Image Editing Models (CVPR 2026 · similarity 0.90)
- Instruction-based Image Manipulation by Watching How Things Move (CVPR 2025 · similarity 0.89)
- Text-Vision Co-Instructed Image Editing (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.