Research map / Vision-language and multimodal / Multimodal LLMs

GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI

NeurIPS 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 15 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces GMAI-MMBench, a 26K-question VQA benchmark built from 284 medical datasets across 38 modalities, organized in a lexical tree by task, department, and perceptual granularity, and evaluates 50 LVLMs zero-shot.

Problem

Existing medical multimodal benchmarks are narrow in modality/task, built from literature-derived image-text pairs, lack clinical relevance, and are not categorized well enough for customized evaluation across clinical tasks, departments, and perceptual granularities.

Approach

Converts curated clinical image-label datasets (public and hospital) into VQA items using a lexical tree that tags each item by modality, clinical VQA task, department, and perceptual granularity (image, box, mask, contour), enabling customized subset evaluation.

Key results

• GPT-4o achieves the best overall test accuracy of 53.96% on GMAI-MMBench, while the random baseline is 25.94%.
• Open-source MedDr (43.69%) and DeepSeek-VL-7B (43.43%) approach proprietary GPT-4V (44.08%) and surpass Claude3-Opus (32.44%) and Qwen-VL-Max (42.16%) on overall test accuracy.
• Box-level perception is consistently the worst granularity, even below image-level, indicating LVLM sensitivity to bounding-box prompts.

Measured effects reported in the paper

Surprising findings

• Box-level annotation yields worse accuracy than unannotated image-level questions across models.
• Most medical-specific models underperform general LVLMs; LLaVA-Med scores below LLaVA-V1.5-7B.

Limitations

• Manual curation may introduce biases and focuses only on medical subjects, which may not fully capture general medical AI.
• Evaluation is zero-shot only and does not include human expert comparison (planned future work).

Experimental setup

Datasets: 284 medical image datasets (public and hospital-shared)

Benchmarks: GMAI-MMBench

Metrics: accuracy macro-averaged accuracy multi-choice accuracy (ACC_mcq) recall (Recall_mcq)

Baselines: GPT-4o GPT-4V Gemini 1.5 Gemini 1.0 Claude3-Opus Qwen-VL-Max MedDr LLaVA-Med Med-Flamingo RadFM Qilin-Med-VL-Chat DeepSeek-VL-7B

Builds on: SA-Med2D-20M protocol VLMEvalKit

Closest papers by meaning

• GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and a Comprehensive Multimodal Dataset Towards General Medical AI (AAAI 2026 · similarity 0.94)
• OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.91)
• How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.90)
• MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models (ICLR 2025 · similarity 0.90)
• Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation (NeurIPS 2025 · similarity 0.89)
• Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks (ICLR 2026 · similarity 0.89)