Research map / Vision-language and multimodal / Multimodal LLMs
MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMIE, a 20K-query knowledge-intensive benchmark for interleaved image-text comprehension and generation in LVLMs, with an InternVL-2-4B-based fine-tuned scoring model (MMIE-Score) for automated evaluation.
Problem
Existing benchmarks for interleaved multimodal comprehension and generation are small, narrow in scope, and rely on costly or biased automated metrics (e.g., GPT-4 judges, CLIPScore), so LVLMs lack reliable evaluation.
Approach
A large, multi-domain interleaved benchmark (situational analysis, project-based learning, multi-step reasoning) paired with a rubric-based scoring model fine-tuned from InternVL-2-4B on 800 human-scored responses, offered as an alternative to GPT-4o judging.
Key results
- Best integrated model GPT-4o + SDXL reaches 65.47% average on MMIE; the average over four open-source interleaved LVLMs is 50.80%, with Anole best at 55.22%.
- The fine-tuned MMIE-Score reaches cosine similarity 0.873 and MAE 1.444 with human scores, outperforming CLIPScore, InternVL-2.0-4B, Anole, and GPT-4o as judges.
- Integrated LVLMs (LVLM + T2I) outperform open-source interleaved LVLMs by an average of 25.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Integrate non-interleaved LVLMs with text-to-image models | Interleaved LVLMs on MMIE · MMIE | average score: 25.2% | Helped |
| Use Anole instead of MiniGPT-5 | Interleaved LVLMs · MMIE | average score: 8.4% | Helped |
| Use Anole instead of GILL | Interleaved LVLMs · MMIE | average score: 7.0% | Helped |
| Use Anole instead of EMU-2 | Interleaved LVLMs · MMIE | average score: 21.8% | Helped |
| Fine-tune InternVL-2-4B on human-annotated scoring data | MMIE-Score · MMIE human-score alignment | cosine similarity: 0.805 -> 0.873 | Helped |
| Generate images alongside text (ablation) | GPT-4o + SDXL · MMIE (100 samples) | average score: 53.46 -> 65.47 | Helped |
| Generate images alongside text (ablation) | GPT-4o + SDXL, GPT-4o evaluation · MMIE (100 samples) | average score: 60.90 -> 71.24 | Helped |
Surprising findings
- Using GPT-4o as a judge gives higher scores than MMIE-Score for the same outputs (e.g., 71.24 vs 65.47 for GPT-4o + SDXL with images), and the Pearson correlations of all judges with humans are low (0.04-0.11).
- Removing image generation from GPT-4o + SDXL lowers the score, supporting that the generated images contribute to quality under the criteria.
Limitations
- Scoring model fine-tuned on a small dataset (800 examples) and may generalize poorly to other tasks; Pearson correlations with human scores are low.
- Only 3.4% of questions are multiple-choice and most are open-ended, and the scoring model is tied to InternVL-2-4B.
- Scoring is rubric-based on six criteria and may not capture all quality aspects; the authors note limited OOD generalization.
Experimental setup
Datasets: VIST Wikihow-VGSI MathVista ReMI COCO
Benchmarks: MMIE
Metrics: MMIE-Score cosine similarity MSE MAE Pearson correlation accuracy
Baselines: MiniGPT-5 EMU-2 GILL Anole GPT-4o Gemini-1.5 LLaVA-v1.6-34b Qwen2-VL-72b
Builds on: InternVL-2 CLIPScore MathVista VIST
Closest papers by meaning
- MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.93)
- MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models (ICLR 2025 · similarity 0.92)
- Learning Interleaved Image-Text Comprehension in Vision-Language Large Models (ICLR 2025 · similarity 0.92)
- WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation (CVPR 2026 · similarity 0.92)
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (ICML 2024 · similarity 0.91)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.