Research map / Vision-language and multimodal / Multimodal LLMs
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MM-Vet, a benchmark of 218 open-ended questions over 200 images that tests six core vision-language capabilities and their 16 integrations, scored by a GPT-4 few-shot LLM evaluator.
Problem
Existing VL benchmarks test one or two capabilities in isolation and use closed-form metrics, so they do not systematically evaluate the integrated, open-ended capabilities of large multimodal models on complicated tasks.
Approach
Defines six core VL capabilities (recognition, OCR, knowledge, language generation, spatial awareness, math) whose combinations yield 16 integrated task types, and scores open-ended answers with a few-shot GPT-4 evaluator that outputs a soft 0-1 score, enabling per-capability and per-integration insights.
Key results
- GPT-4V achieves the highest total score of 67.7% on MM-Vet, well above open-source LMMs.
- MM-ReAct-GPT-4 (tool-using) reaches 44.6% total, with the best OCR (65.7%), spatial (56.8%) and math (69.2%) scores among its comparisons.
- Among end-to-end open LMMs, LLaVA-13B (LLaMA-2) scores 32.9% total; GPT-4 as evaluator has mean absolute difference 0.042 from human scores versus 0.273 for keyword matching.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4V instead of MM-ReAct-GPT-4 | MM-Vet evaluation · MM-Vet | Total score: 44.6$\pm$0.2 -> 67.7$\pm$0.3 | Helped |
| Use LLaVA-13B (V1.3, 336px) instead of LLaVA-13B (LLaMA-2) | LLaVA · MM-Vet, Recognition and spatial awareness integration | score: +8.4% | Helped |
| Replace GPT-3.5 with GPT-4 in MM-ReAct | MM-ReAct · MM-Vet, OCR and spatial awareness integration | score: over 40% | Helped |
| Replace GPT-3.5 with GPT-4 in MM-ReAct | MM-ReAct · MM-Vet, OCR-only integration | score: 23.0% | Helped |
| Add totally right/wrong short-answer grading examples (ablation) | GPT-4 evaluator · MM-Vet evaluator human-score agreement | Average absolute difference: 0.0630 -> 0.0625 | Helped |
| Add partially right short-answer grading examples (ablation) | GPT-4 evaluator · MM-Vet evaluator human-score agreement | Average absolute difference: 0.0625 -> 0.0619 | Helped |
| Add a partially right and partially wrong short-answer example (ablation) | GPT-4 evaluator · MM-Vet evaluator human-score agreement | Average absolute difference: 0.0619 -> 0.0551 | Helped |
| Add a fully right long-answer grading example (ablation) | GPT-4 evaluator · MM-Vet evaluator human-score agreement | Average absolute difference: 0.0551 -> 0.0520 | Helped |
| Add a partially right long-answer grading example (ablation) | GPT-4 evaluator · MM-Vet evaluator human-score agreement | Average absolute difference: 0.0520 -> 0.0423 | Helped |
Surprising findings
- Keyword matching is far less aligned with human grading (difference 0.273) than the GPT-4 evaluator, and weaker LLM judges such as the largest LLaMA-2 and Mistral models deviate more from human scores.
- Combining GPT-4 with other LLMs in the evaluator did not reduce disagreement with humans; GPT-4 alone performed best.
Limitations
- Only image-text input and text output are covered; other modalities are not evaluated.
- Relies on GPT-4 as the grader, which incurs API costs; only GPT-4 matched human grades well among the judges tested.
- Small benchmark (218 samples over 200 images) gives limited per-integration sample sizes.
Experimental setup
Datasets: VCR ChestX-ray14
Benchmarks: MM-Vet
Metrics: LLM-graded score (0-1, averaged to %) absolute difference to human scores
Baselines: OpenFlamingo-9B BLIP-2-12B LLaVA-13B (LLaMA-2) MiniGPT-4 LLaMA-Adapter v2-7B Otter-9B InstructBLIP-8B MM-ReAct-GPT-4 Transformers Agent GPT-4V Bard
Builds on: LLaVA MM-ReAct BLIP-2 InstructBLIP
Closest papers by meaning
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.91)
- MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency (ICML 2025 · similarity 0.90)
- MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos (ICLR 2026 · similarity 0.90)
- UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation (CVPR 2025 · similarity 0.90)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.90)
- MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.