Research map / Vision-language and multimodal / Multimodal LLMs

MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 119 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MME, a manually annotated yes/no benchmark with 14 perception and cognition subtasks, and evaluates 30 MLLMs zero-shot to expose problems like instruction-following failures and object hallucination.

Problem

Existing MLLM evaluations rely on traditional datasets that may leak into training, use open-ended answers hard to score, or cover only one aspect; a comprehensive, leakage-resistant benchmark with objective scoring is needed.

Approach

Each image gets two manually written yes/no questions (answer yes and no), with a fixed suffix "Please answer yes or no," scored by accuracy and accuracy+ (both correct per image), enabling objective, prompt-agnostic comparison across 14 perception and cognition subtasks.

Key results

• GPT-4V leads cognition tasks with top scores on code reasoning (170) and numerical calculation, while none of the models exceed 150 on commonsense, numerical, or translation subtasks.
• Many models achieve near-chance accuracy+ on position and count, and object hallucination yields about 50% accuracy and near 0 accuracy+ when models constantly answer yes.
• GPT-4V scores 185 on OCR, leading Skywork-MM (162.5) and WeMM (147.5).

Measured effects reported in the paper

Surprising findings

• GPT-4V refuses to answer celebrity questions, receiving a zero score on that subtask.
• Models that answer 'yes' to everything reach about 50% accuracy but near 0 accuracy+, revealing hallucination.

Limitations

• Tasks are relatively simple in this version, especially OCR, numerical, translation and code reasoning.
• Yes/no format limits open-ended response assessment and may not reflect multiple-choice or generative abilities.
• Some public image sources (COCO, fine-grained datasets) are reused, so leakage cannot be fully excluded.

Experimental setup

Datasets: COCO ScienceQA

Benchmarks: MME

Metrics: accuracy accuracy+

Baselines: BLIP-2 InstructBLIP MiniGPT-4 LLaVA GPT-4V Qwen-VL-Chat mPLUG-Owl2 SPHINX

Builds on: COCO ScienceQA

Closest papers by meaning

• MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans? (ICLR 2025 · similarity 0.92)
• Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark (NeurIPS 2025 · similarity 0.92)
• MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs (ICLR 2025 · similarity 0.91)
• SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.91)
• On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.90)
• MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.90)