Research map / Vision-language and multimodal / Multimodal LLMs
MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMIU, a 11.7K-question, 52-task multiple-choice benchmark for multi-image understanding in LVLMs, and evaluates 24 LVLMs, finding GPT-4o best at 55.7%.
Problem
Existing multi-image evaluation benchmarks for large vision-language models cover few image relationships and tasks, so LVLMs' multi-image comprehension is not comprehensively measured.
Approach
A top-down benchmark built from 7 image relationships (semantic, temporal, spatial) grounded in cognitive psychology memory theory, with 52 tasks assembled from existing datasets, answerable and unanswerable question sets, and analyses via performance by relationship, task map, and SFT learning difficulty.
Key results
- GPT-4o achieves the highest overall accuracy at 55.7% on MMIU, with Claude3.5-Sonnet and Gemini1.5 at 53.4%.
- InternVL2 is the best open-source model at 50.3%, about 5.4 points below GPT-4o.
- Spearman correlation between SFT accuracy and model accuracy across tasks is 0.66.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply multi-image SFT to Idefics2 | Idefics2 / Mantis · MMIU | accuracy: +17.8% accuracy | Helped |
| include unanswerable questions (ablation) | GPT4o · MMIU GAR | accuracy | Hurt |
| include unanswerable questions (ablation) | InternVL1.5 · MMIU GAR | accuracy | Hurt |
| concatenate visual tokens instead of concatenating images (ablation) | single-image LVLMs · MMIU | accuracy | Hurt |
| concatenate images instead of visual tokens (ablation) | LLaVA series · MMIU | accuracy | Helped |
| switch between image stitching and visual-token stitching (ablation) | GLM-4V · MMIU | accuracy | No clear effect |
Surprising findings
- InternVL1.5-Chat, trained only on single-image data, (37.4%) outperforms several multi-image models such as LLaVA-Interleave and Idefics2.
- Concatenating visual tokens for single-image models generally did not beat concatenating raw images.
Limitations
- Single-image models are evaluated via image concatenation, which may not reflect their native multi-image capability.
- Some closed-source samples were discarded due to copyright refusals.
Experimental setup
Datasets: RecipeQA NLVR Kinetics MSCOCO ScanNet BLINK GUI-Odyssey RAVEN Market-1501 LFW NeXTQA MSVD
Benchmarks: MMIU
Metrics: accuracy
Baselines: GPT-4o Claude3.5-Sonnet Gemini1.5 Gemini1.0 Pro Vision InternVL2 Mantis LLaVA-Interleave InternVL1.5-Chat GLM-4V-9B Idefics2-8B DeepSeek-VL-7B Qwen-VL-Chat
Builds on: MileBench MUIRBench Video-MME MMT-Bench
Closest papers by meaning
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.92)
- MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence (ICLR 2026 · similarity 0.92)
- MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.92)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.91)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.91)
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.