Research map / Image generation / Editing and style transfer
I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes I2EBench, an automatic benchmark of 2,000+ images and 4,000+ instructions across 16 dimensions (8 high-level, 8 low-level) for evaluating instruction-based image editing models, with human-ranking correlation analysis.
Problem
Existing evaluation of instruction-based image editing relies on conventional metrics or small, narrow benchmarks that cannot cover diverse editing types and align with human perception.
Approach
Use dimension-specific automatic evaluators (GPT-4V question answering against human-annotated answers, CLIP similarity for style, SSIM for low-level and region accuracy) across 16 editing dimensions, and validate the resulting scores against human rankings.
Key results
- No single IIE model dominates all 16 dimensions; InstructDiffusion leads 4 of 7 low-level dimensions with original instructions, and MagicBrush leads 3 high-level dimensions with original instructions.
- MGIE scores 11.92 points above the second-place model on deblurring with original instructions.
- I2EBench rank scores show significant positive correlation with human preference rankings across all dimensions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use diverse rather than original instructions | MGIE · Deblurring | I²EBench score: 60.30 -> 57.33 | Hurt |
| Use diverse rather than original instructions | InstructDiffusion · Noise Removal | I²EBench score: 28.60 -> 28.06 | Hurt |
| Use diverse rather than original instructions | InstructDiffusion · Watermark Removal | I²EBench score: 85.49 -> 85.14 | Hurt |
| Use diverse rather than original instructions | MagicBrush · Counting | I²EBench score: 30.71 -> 24.29 | Hurt |
| Use diverse rather than original instructions | MagicBrush · Direction Perception | I²EBench score: 49.29 -> 45.71 | Hurt |
| Use diverse rather than original instructions | InstructAny2Pix · Direction Perception | I²EBench score: 41.73 -> 51.80 | Helped |
| Use diverse rather than original instructions | InstructAny2Pix · Object Removal | I²EBench score: 46.76 -> 43.88 | Hurt |
| Use diverse rather than original instructions | HIVE · Background Replacement | I²EBench score: 74.29 -> 85.00 | Helped |
| Use diverse rather than original instructions | InstructEdit · Region Accuracy | I²EBench score: 77.08 -> 77.08 | No clear effect |
| Use diverse rather than original instructions | MagicBrush · Color Alteration | I²EBench score: 55.71 -> 54.29 | No clear effect |
Surprising findings
- Scenery and Global categories consistently score higher than other categories across all evaluated models.
- Models using LLM or MLLM instruction encoders show more stable performance across original versus diverse instructions.
Limitations
- Relies on GPT-4V as an automatic judge for most high-level dimensions, inheriting its errors and cost.
- Human evaluation is limited in scale; correlation is computed over ranks among a small set of models (8).
Experimental setup
Datasets: public image editing datasets (unspecified)
Benchmarks: I2EBench
Metrics: GPT-4V answer agreement CLIP similarity SSIM human rank score correlation
Baselines: HIVE InstructDiffusion InstructPix2Pix MagicBrush MGIE InstructEdit InstructAny2Pix HQ-Edit
Builds on: InstructPix2Pix MagicBrush
Closest papers by meaning
- ImgEdit: A Unified Image Editing Dataset and Benchmark (NeurIPS 2025 · similarity 0.90)
- Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing (ICML 2025 · similarity 0.90)
- EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing (ICLR 2026 · similarity 0.90)
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.89)
- SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models (CVPR 2024 · similarity 0.88)
- EditBoard: Towards a Comprehensive Evaluation Benchmark for Text-Based Video Editing Models (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.