Research map / Vision-language and multimodal / Multimodal LLMs
MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MuirBench, a 2,600-question multiple-choice benchmark with paired answerable/unanswerable multi-image instances across 12 tasks and 10 multi-image relations, evaluating 20 multimodal LLMs.
Problem
Existing multimodal LLM benchmarks mostly test single-image understanding and lack comprehensive, robust evaluation of reasoning across multiple images.
Approach
A benchmark pairing each answerable multi-image question with a minimally changed unanswerable counterpart (image replacement/reordering, question modification, or option replacement), spanning 12 tasks and 10 multi-image relation categories.
Key results
- GPT-4o achieves 68.0% overall accuracy on MuirBench, versus 93.15% human performance.
- Gemini Pro reaches 49.3% accuracy, while open-source multi-image models hover between 23.73% and 44.50%.
- Open-source models trained on single images hover below 33.3% accuracy, and models perform much worse on unanswerable instances (GPT-4o gap of 26.8%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-image-input models rather than models designed for single-image input | Multimodal LLMs evaluated on MuirBench · MuirBench overall | accuracy | Helped |
| Evaluate models on unanswerable counterparts instead of answerable instances | Multimodal LLMs · MuirBench answerable vs unanswerable instances | accuracy: 26.8% | Hurt |
| Use unanswerable counterparts | GPT-4o · MuirBench answerable vs unanswerable instances | accuracy: 26.8% | Hurt |
| Use unanswerable counterparts | Gemini Pro · MuirBench answerable vs unanswerable instances | accuracy: 21.5% | Hurt |
| Place images in the middle of questions rather than in options | GPT-4o, Gemini Pro, and Mantis-8B-Idefics2 · MuirBench image-position analysis | accuracy | Hurt |
| Remove the correct option | GPT-4o, Gemini Pro, and Mantis-8B-Idefics2 · MuirBench unanswerable instances | accuracy | Hurt |
| Modify the question to make it incompatible with the original images and options | GPT-4o, Gemini Pro, and Mantis-8B-Idefics2 · MuirBench unanswerable instances | accuracy | Helped |
Surprising findings
- No obvious correlation between model size and performance on multi-image understanding.
- Some models perform at or below random guessing on certain tasks.
Limitations
- Focuses only on 2D images, not 3D problems.
- Unanswerable construction strategies do not cover all possible strategies.
- All tasks are in English, which may introduce bias in multilingual settings.
Experimental setup
Datasets: GeneCIS SEED-Bench IconQA NLVR2 HallusionBench ISVQA MMBench National Geologic Map Database University-1652 PubMed SciDuet
Benchmarks: MuirBench
Metrics: accuracy
Baselines: GPT-4-Turbo Gemini Pro Mantis-8B-Idefics2 Mantis-8B-clip-llama3 Mantis-8B-siglip-llama3 Idefics-9B-Instruct Idefics2-8B Emu2-Chat VILA1.5-13B OpenFlamingo-v2-9B LLaVA-NeXT-34B LLaVA-v1.5
Builds on: MMBench HallusionBench VLMEvalKit MathVista prompt format MMMU evaluation tool
Closest papers by meaning
- MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models (ICLR 2025 · similarity 0.90)
- JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images (NeurIPS 2024 · similarity 0.89)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.88)
- Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark (ICLR 2025 · similarity 0.88)
- Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping (ICLR 2025 · similarity 0.88)
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.