Research map / Vision-language and multimodal / Multimodal LLMs
FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
FireEdit adds region tokens from a Deformable DETR detector to a LLaVA-7B-based VLM and uses time-aware target injection and hybrid visual cross-attention to guide a Stable Diffusion 1.5 editor for instruction-based image editing.
Problem
Instruction-based image editing struggles with complex scenes, semantic consistency, and fine-grained localization, since CLIP text encoders and T2I-synthesized training data fail to ground instructions to the correct regions.
Approach
Inject detector-derived region tokens into the VLM input so it can localize editing targets, then guide the diffusion model with time-dependent instruction features (TATI) and multi-scale visual features (HVCA, CLIP+DINOv2) to preserve non-edited content.
Key results
- On the Emu Edit test set, FireEdit reaches L1 0.0574, CLIP-I 0.9140, DINO 0.8829 and LPIPS 0.1373, the best among compared methods, and receives 78.67% user preference.
- On MagicBrush single-turn, FireEdit gets L1 0.0701 and CLIP-I 0.9131, beating SmartEdit (0.0936 / 0.8945) but not MagicBrush's own model on L1 (0.0625) or DINO (0.8987 vs 0.8619 for FireEdit).
- Ablation on Emu Edit: adding region tokens lowers L1 from 0.0897 to 0.0859 and raises CLIP-I from 0.8501 to 0.8709; adding TATI further lowers L1 to 0.0609 and raises CLIP-I to 0.9258.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add region-aware mixed-modal encoding (region tokens) (ablation) | FireEdit · Emu Edit test | L1: 0.0897 -> 0.0859 | Helped |
| add region-aware mixed-modal encoding (region tokens) (ablation) | FireEdit · Emu Edit test | CLIP-I: 0.8501 -> 0.8709 | Helped |
| add TATI (ablation) | FireEdit with Region · Emu Edit test | L1: 0.0859 -> 0.0609 | Helped |
| add TATI (ablation) | FireEdit with Region · Emu Edit test | CLIP-I: 0.8709 -> 0.9258 | Helped |
| add HVCA (ablation) | FireEdit with Region and TATI · Emu Edit test | L1: 0.0609 -> 0.0574 | Helped |
| add HVCA (ablation) | FireEdit with Region and TATI · Emu Edit test | LPIPS: 0.1580 -> 0.1373 | Helped |
| add HVCA (ablation) | FireEdit with Region and TATI · Emu Edit test | CLIP-T: 0.2746 -> 0.2783 | Helped |
| use FireEdit instead of MagicBrush | instruction-based image editing · MagicBrush test single-turn | L1: 0.0625 -> 0.0701 | Hurt |
| use FireEdit instead of MagicBrush | instruction-based image editing · MagicBrush test single-turn | CLIP-I: 0.9332 -> 0.9131 | Hurt |
| use FireEdit instead of MagicBrush | instruction-based image editing · MagicBrush test multi-turn | L1: 0.0964 -> 0.0911 | Helped |
| use FireEdit instead of MagicBrush | instruction-based image editing · MagicBrush test multi-turn | CLIP-I: 0.8924 -> 0.8819 | Hurt |
Surprising findings
- MagicBrush's own model outperforms FireEdit on several single-turn MagicBrush metrics (L1 0.0625 vs 0.0701, DINO 0.8987 vs 0.8619), which the authors attribute to biases from its training set.
- The paper reports that the CLIP-T score on Emu Edit does not improve with region tokens alone (0.2770 to 0.2762).
Limitations
- Does not support reference images as input.
- Focuses on simple text instructions; pose changes and object repositioning are noted as challenging.
Experimental setup
Datasets: COCO-Stuff RefCOCO GRefCOCO InstructPix2Pix MagicBrush UltraEdit ReasonEdit LLaVA-instruct-150k
Benchmarks: Emu Edit test MagicBrush test
Metrics: L1 L2 CLIP-I DINO LPIPS CLIP-T user study
Baselines: InstructPix2Pix MagicBrush HIVE HQ-Edit UltraEdit MGIE SmartEdit SDEdit NTI GLIDE BleDiff
Builds on: InstructPix2Pix LLaVA IP-Adapter Deformable DETR SmartEdit
Closest papers by meaning
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.91)
- SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models (CVPR 2024 · similarity 0.90)
- UltraEdit: Instruction-based Fine-Grained Image Editing at Scale (NeurIPS 2024 · similarity 0.90)
- AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea (CVPR 2025 · similarity 0.89)
- ImgEdit: A Unified Image Editing Dataset and Benchmark (NeurIPS 2025 · similarity 0.89)
- Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.