Research map / Vision-language and multimodal / Multimodal LLMs
MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMIG-Bench, a unified benchmark of 4,850 prompts and 1,750 multi-view reference images for text-only and image-text-conditioned image generation, with a three-level evaluation suite including the VQA-based Aspect Matching Score (AMS).
Problem
Multi-modal image generators (e.g., GPT-4o, Gemini) that accept both text and reference images are evaluated with disjoint toolkits: T2I benchmarks lack multimodal conditioning, and customization benchmarks are small and overlook compositional semantics and common knowledge.
Approach
Unify T2I and customization evaluation in one benchmark with grouped multi-view reference images and dense prompt labels, and score generations at three levels (low-level artifacts/identity, mid-level VQA-based AMS, high-level aesthetics/preference).
Key results
- AMS achieves Spearman rho 0.699 with human ratings, above DSG (0.692), VQ2 (0.399), and BLIPVQA (0.147).
- HiDream-I1-Full ranks first on most T2I metrics (AMS 89.65, HPSv2 0.321, PickScore 0.450).
- In customization, DreamBooth gives the strongest identity preservation, while GPT-4o achieves the top AMS (90.90) and aesthetic score (6.368) but weaker identity scores.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AMS instead of DSG for automated evaluation | T2I metric evaluation · T2I human alignment | Spearman correlation: 0.692 -> 0.699 | Helped |
| Use AMS instead of VQ2 for automated evaluation | T2I metric evaluation · T2I human alignment | Spearman correlation: 0.399 -> 0.699 | Helped |
| Use AMS instead of BLIPVQA for automated evaluation | T2I metric evaluation · T2I human alignment | Spearman correlation: 0.147 -> 0.699 | Helped |
| Use HiDream-I1-Full rather than other evaluated T2I generators | T2I image generation · MMIG-Bench T2I | AMS: 89.65 | Helped |
| Use GPT-4o rather than the evaluated diffusion models | Multimodal image generation · MMIG-Bench customization | AMS: 90.90 | Helped |
| Use DreamBooth rather than other evaluated customization models | Customization generation · MMIG-Bench customization | DINOv2: 38.961 | Helped |
| Use DreamBooth rather than other evaluated customization models | Customization generation · MMIG-Bench customization | AMS: 28.00 | Hurt |
| Use GPT-4o rather than DreamBooth | Customization generation · MMIG-Bench customization | CLIP-I: 88.758 vs 75.152 | Hurt |
| Scale up the image generation model | T2I image generation · MMIG-Bench T2I | AMS | Helped |
Surprising findings
- GPT-4o cannot preserve identity well, performing worse than early models like Emu2 and encoder-based BLIP Diffusion and IP-Adapter on identity metrics.
- DreamBooth, despite strongest identity preservation, often fails to generate the correct scene, actions, or additional entities.
Limitations
- Human ratings do not cover visual quality dimensions.
- Counting aspect is rare in the customization task (0.1% of QA pairs).
- Relies on GPT-4o for prompt and QA generation and Qwen-VL2.5-72B as VQA judge.
Experimental setup
Datasets: PartiPrompts Gecko T2I-CompBench++ GenEval GenAI-Bench T2I-FactualBench FineMatch
Benchmarks: MMIG-Bench
Metrics: CLIP-T CLIP-I DINOv2 CUTE PAL4VST AMS BLIPVQA Aesthetic HPSv2 PickScore
Baselines: SDXL Photon-v1 Lumina-2 HunyuanDiT-v1.2 PixArt-Sigma FLUX.1-dev SD 3.5-large HiDream-I1-Full JanusFlow Janus-Pro-7B Gemini-2.0-Flash GPT-4o
Builds on: FineMatch DSG VQ2 BLIPVQA DreamBooth IP-Adapter
Closest papers by meaning
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.91)
- MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? (NeurIPS 2025 · similarity 0.90)
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.90)
- DuoGen: Towards Autonomous Interleaved Multimodal Generation (CVPR 2026 · similarity 0.89)
- SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation (CVPR 2025 · similarity 0.89)
- A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.