Research map / Vision-language and multimodal / Multimodal LLMs
EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EdiVal-Agent, an object-centric automated framework that generates multi-turn editing instructions from decomposed image objects and scores instruction following, content consistency, and visual quality, then benchmarks 13-16 editors on EdiVal-Bench.
Problem
Evaluating instruction-based image editing, especially multi-turn editing, is unreliable: reference-based metrics are limited and biased, while VLM-only judges are imprecise for spatial reasoning, small pixel changes, and artifacts.
Approach
Decompose images into objects with a VLM, generate multi-turn instructions while tracking evolving object pools, then score edits with detector-based symbolic checks plus VLM verification on detector crops (EdiVal-IF), DINO-feature consistency of background and unchanged objects (EdiVal-CC), and a human preference model (HPSv3) for visual quality.
Key results
- Seedream 4.0 ranks first overall with EdiVal-O of 83.81/69.95/59.76 at turns 1-3.
- EdiVal-IF reaches 81.3% human agreement, versus 75.2% for Qwen2-VL and 65.4% for thresholded CLIP_dir.
- Qwen-Image-Edit degrades sharply across turns (EdiVal-O 78.36 -> 59.56 -> 41.93), which the authors attribute to exposure bias.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Grounding DINO detector with OWL-ViT in EdiVal-IF (ablation) | EdiVal-IF · human agreement study | human agreement accuracy: 81.30% to 53.67% | Hurt |
| Use EdiVal-IF (detector plus VLM on crops) instead of VLM alone | instruction-following evaluation · human agreement study | human agreement accuracy: 81.3% vs 75.2% (Qwen2-VL alone) | Helped |
| Use EdiVal-IF instead of thresholded CLIP_dir for instruction following | instruction-following evaluation · human agreement study | human agreement accuracy: 81.3% vs 65.4% | Helped |
| Swap default VLM (Qwen2-7B-VL) for Qwen2.5-32B-VL (ablation) | EdiVal-IF · VLM swap correlation | Pearson correlation of per-turn IF success rates: 0.9790 Pearson | No clear effect |
| Replace Grounding DINO with OWL-ViT detector (ablation) | EdiVal-IF · per-turn success correlation | Pearson correlation with Grounding DINO results: Pearson 0.82, absolute scores drop | Hurt |
| Change Grounding DINO threshold (0.3 or 0.4 global) or disable large-box filter (ablation) | EdiVal-IF · Img/Task success, overall score | overall score ranking and values: small changes; ranking unchanged | No clear effect |
| Swap DINOv3 feature extractor for DINOv2 in EdiVal-CC (ablation) | EdiVal-CC · per-turn CC ranking | Pearson correlation of per-turn CC scores: Pearson 0.9987, Spearman 1.0000; absolute values lower | No clear effect |
| Upgrade from GPT-Image-1 to GPT-Image-1.5 | GPT-Image series · EdiVal-Bench multi-turn | EdiVal-CC and EdiVal-O: EdiVal-CC +13.49/+13.42/+12.99 across turns | Helped |
| Upgrade from Nano Banana to Nano Banana 2 | Nano series · EdiVal-Bench multi-turn | EdiVal-O: 81.48/67.70/56.24 to 83.14/70.02/58.49 | Helped |
| Use Qwen-Image-Edit multi-turn vs single-shot complex prompt | Qwen-Image-Edit · EdiVal-Bench multi-turn | EdiVal-O across turns: 78.36 to 41.93 by turn 3 | Hurt |
| Compress three-turn instructions with keep-unchanged constraint instead of default concatenation (ablation) | Qwen-Image-Edit complex C3 editing · complex editing success rate | success rate (%): 27.62 to 25.87 | Hurt |
| Random shuffle or sequential connector for compressed prompts (ablation) | Qwen-Image-Edit complex C3 editing · complex editing success rate | success rate (%): 27.10 and 26.92 vs 27.62 | No clear effect |
Surprising findings
- GPT-Image-1 achieves strong instruction following but ranks near the bottom on content consistency.
- Replacing Grounding DINO with OWL-ViT drops EdiVal-IF human agreement from 81.3% to 53.67%.
Limitations
- Instruction types are limited to object-centric edits; style_change is excluded.
- Evaluation relies on detector accuracy; Grounding DINO false positives and misses cause evaluation errors.
- The framework does not use evaluation scores to improve editing models.
Experimental setup
Datasets: GEdit-Bench
Benchmarks: EdiVal-Bench
Metrics: EdiVal-IF EdiVal-CC EdiVal-VQ EdiVal-O human agreement accuracy
Baselines: GPT-Image-1 Nano Banana FLUX.1-Kontext-max FLUX.1-Kontext-dev Qwen-Image-Edit Step1X-Edit OmniGen AnyEdit UltraEdit MagicBrush InstructPix2Pix Qwen2-VL
Builds on: GEdit-Bench Grounding DINO HPSv3 DINOv3 InstructPix2Pix
Closest papers by meaning
- Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing (ICML 2025 · similarity 0.91)
- I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing (NeurIPS 2024 · similarity 0.90)
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.90)
- ImgEdit: A Unified Image Editing Dataset and Benchmark (NeurIPS 2025 · similarity 0.90)
- Learning Action and Reasoning-Centric Image Editing from Videos and Simulation (NeurIPS 2024 · similarity 0.89)
- Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.