Research map / Vision-language and multimodal / Multimodal LLMs

MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MME-RealWorld, a fully human-annotated, high-resolution benchmark (29,429 QA pairs, 43 subtasks, 5 real-world domains) plus a Chinese variant, showing that 29 MLLMs all score below 60% accuracy.

Problem

Existing MLLM benchmarks are small, rely on model-generated annotations, and are too easy because of low image resolution, so they fail to expose real-world perception and reasoning limits and produce high variance.

Approach

A large, fully manually annotated multiple-choice benchmark built from very high-resolution real-world images (avg ~2000x1500, up to 5304x7952) across OCR, remote sensing, diagrams/tables, autonomous driving, and monitoring, with hard distractors and an extra "E" reject option, plus a separately collected Chinese split built from Chinese-scene images rather than translation.

Key results

• Across 29 evaluated MLLMs, none reaches 60% accuracy on MME-RealWorld; the top perception average is InternVL-2/Qwen2-VL near 55-59%, and reasoning averages top out near 46%.
• LLaVA-1.5-7B scores 24.9% overall, the lowest difficulty baseline among compared benchmarks.
• Qwen2-VL reaches 59.80% average perception accuracy on the Chinese split, above its English split score.

Measured effects reported in the paper

Surprising findings

• Closed-source models such as GPT-4o often choose the reject option "E" far more frequently than its true frequency (under 5%), indicating conservative behavior and failure to find objects.
• Gemini 1.5 Pro scores only 11.14% average on the Chinese reasoning split, far below open-source peers.

Limitations

• Closed-source models face image size and upload limits, forcing compression of high-resolution inputs that may confound results.
• Models largely fail on dynamic information (driving, monitoring) even though inputs are single frames.

Experimental setup

Datasets: DIV2K Flickr2K LIU4K UHD4K UHD8K HQ-50K FAIR1M VisDrone nuScenes CODA Rank2Tell LLVIP

Benchmarks: MME-RealWorld MME-RealWorld-CN

Metrics: accuracy average accuracy class-based average accuracy

Baselines: GPT-4o Gemini 1.5 Pro Claude 3.5 Sonnet InternVL-2 Qwen2-VL LLaVA-1.5 Mini-Gemini-HD SliME Cambrian-1

Builds on: MME MMBench SEED-Bench MMT-Bench

Closest papers by meaning

• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (NeurIPS 2025 · similarity 0.92)
• SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.91)
• MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.90)
• On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.90)
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.90)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)