Research map / Vision-language and multimodal / Multimodal LLMs
Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RISEBench, a 360-sample human-curated benchmark for reasoning-informed visual editing across temporal, causal, spatial, and logical reasoning, scored by GPT-4.1 as judge with human validation.
Problem
Existing visual editing models struggle with complex reasoning-based instructions, and no established benchmark systematically evaluates reasoning-informed visual editing (RISE).
Approach
A focused benchmark of 360 human-annotated editing cases spanning four reasoning categories, with a three-dimension evaluation (instruction reasoning, appearance consistency, visual plausibility) using an LMM-as-a-judge pipeline validated against six human experts.
Key results
- GPT-4o-Image achieves the highest overall accuracy at 28.9% on RISEBench, while most open-source models score near 0%.
- Gemini-2.0-Flash-Experimental scores 13.3% and Gemini-2.0-Flash-Preview 9.4% overall; BAGEL reaches 5.8%.
- LMM-as-a-judge vs human expert MAE is 0.5 (instruction reasoning), 0.7 (appearance consistency), and 0.4 (visual plausibility) on a 1-5 scale.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o-Image instead of Gemini-2.0-Flash-exp | RISEBench visual editing · RISEBench overall | accuracy: 13.3% -> 28.9% | Helped |
| Use Gemini-2.0-Flash-preview instead of Gemini-2.0-Flash-exp | Gemini-2.0-Flash Series · RISEBench Spatial | accuracy: 23.0% -> 11.0% | Hurt |
Surprising findings
- GPT-4o-Image reaches only 10.6% accuracy on logical reasoning despite strong results on other categories.
- Gemini-2.0-Flash-Preview produces higher image quality than Experimental yet scores lower overall due to a drop in spatial reasoning (23.0% to 11.0%).
Limitations
- Only 360 samples, a small benchmark size.
- Task categories may not be exhaustive.
- Judge scores are from GPT-4.1/GPT-4o and human agreement is weaker for intermediate scores.
Experimental setup
Datasets: Images from existing datasets and benchmarks Blender-rendered 3D scenes Internet images under permissive licenses Images generated by image generation models
Benchmarks: RISEBench
Metrics: accuracy (all three dimensions scored 5) instruction reasoning score appearance consistency score visual plausibility score MAE vs human scores
Baselines: GPT-4o-Image Gemini-2.0-Flash-Experimental Gemini-2.0-Flash-Preview BAGEL Step1X-Edit OmniGen EMU2 HiDream-Edit FLUX.1-Canny
Builds on: LMM-as-a-judge
Closest papers by meaning
- KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Models (NeurIPS 2025 · similarity 0.91)
- ReasonEdit: Towards Reasoning-Enhanced Image Editing Models (CVPR 2026 · similarity 0.90)
- Reasoning to Edit: Hypothetical Instruction-Based Image Editing with Visual Reasoning (ICML 2026 · similarity 0.90)
- GIR-Bench: Versatile Benchmark for Generating Images with Reasoning (ICLR 2026 · similarity 0.89)
- EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing (ICLR 2026 · similarity 0.89)
- UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.