Research map / Vision-language and multimodal / Multimodal LLMs
CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CREval, a fully automated QA-based evaluation pipeline (IF/VC/VQ yes/no questions answered by an MLLM), and CREval-Bench (874 creative instruction-image pairs over 9 dimensions) to benchmark image generation and editing models.
Problem
Existing instruction-based image editing benchmarks focus on common edits and rely on opaque MLLM holistic scoring, so they do not systematically or interpretably evaluate complex, creative free-style image manipulation.
Approach
Decompose evaluation into CoT-generated yes/no questions per instruction across instruction following, visual consistency (weighted by element importance), and visual quality, then aggregate the MLLM's answers into a weighted score (0.4 IF, 0.4 VC, 0.2 VQ), with a creative-focused benchmark spanning nine dimensions.
Key results
- Seedream 4.0 achieves the highest overall CREval score (83.43), ahead of Gemini 2.5 Flash Image (81.34) and Qwen-Image-Edit-2509 (79.78) under GPT-4o evaluation.
- Closed-source models lead overall, but all models struggle with visual consistency; GPT-Image-1 has the lowest VC among closed-source models (63.46 overall VC).
- Human scores on six models correlate with CREval ranking; Seedream 4.0 gets the highest HumanScore of 72.01 versus 49.98 for Bagel.
Surprising findings
- Bagel (think) and Step1X-Edit (think) variants score lower than their non-thinking counterparts on instruction following, suggesting the thinking module does not help.
- UniWorld-V1 gets high visual consistency mainly because it fails to execute edits, so unedited outputs score well on VC.
Limitations
- MLLMs are weak at detecting subtle visual artifacts such as extra fingers, so VQ scores are compressed across models.
- Reliance on MLLM judges (GPT-4o, Qwen3-VL) whose absolute scores depend on the evaluator's capability.
- VC rewards unchanged outputs, so consistency metrics can be gamed by failing to edit.
Experimental setup
Datasets: curated images from online resources MagicBrush-style source datasets [5, 14, 31, 37]
Benchmarks: CREval-Bench
Metrics: Instruction Following (IF) Visual Consistency (VC) Visual Quality (VQ) weighted average score human preference alignment
Baselines: VIEScore Aesthetic Score EditScore ImgEdit-Bench KRIS-Bench RISE-Bench GEdit-Bench
Builds on: VIEScore Chain-of-Thought prompting
Closest papers by meaning
- EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing (ICLR 2026 · similarity 0.89)
- SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models (CVPR 2024 · similarity 0.88)
- Guiding Instruction-based Image Editing via Multimodal Large Language Models (ICLR 2024 · similarity 0.88)
- ReasonEdit: Towards Reasoning-Enhanced Image Editing Models (CVPR 2026 · similarity 0.88)
- CreBench: Human-Aligned Creativity Evaluation from Idea to Process to Product (AAAI 2026 · similarity 0.87)
- I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.