Research map / Vision-language and multimodal / Multimodal LLMs

OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OmniBrainBench, a brain-imaging VQA benchmark (9,527 verified pairs, 15 modalities, 15 clinical tasks) and evaluates 24 general, medical, and proprietary MLLMs against physician performance.

Problem

Existing brain imaging VQA benchmarks cover few modalities and only a subset of clinical workflow stages, so MLLMs cannot be comprehensively assessed across the clinical continuum.

Approach

A multi-modality (15 modalities) and multi-stage (five phases, 15 tasks) brain imaging VQA benchmark with closed-ended multiple-choice and open-ended evaluation, built from 30 public sources with GPT-5-assisted question generation and embedding-based deduplication, validated by a radiologist, and benchmarked against physician accuracy.

Key results

• Gemini-2.5-Pro achieves the highest closed-ended overall accuracy among models at 66.58%, versus 91.35% for physicians.
• HuatuoGPT-V-34B reaches 63.56% closed-ended accuracy, the best open-source medical model, while MedGemma-4B reaches 48.04% and LLaVA-Med-7B 38.84%.
• In open-ended evaluation, Lingshu-32B scores highest among open-source models (ROUGE-1 26.45, BLEU 1.90), with Qwen3-VL-30B and GPT-5 also competitive.

Surprising findings

• Lingshu-32B outperforms the much larger HuatuoGPT-V-34B in open-ended generation, suggesting targeted training can beat scale.
• Performance on several tasks (e.g., prognostic factor analysis, postoperative outcome assessment) reaches 100% for some models, while risk stratification remains hard for all.

Limitations

• Benchmark cannot replace final clinical evaluation for safety, per the authors.
• Open-ended evaluation relies on lexical and embedding similarity metrics, which are weakly tied to clinical correctness.
• Data drawn from public and open-access sources, with possible source bias and uneven modality sizes (e.g., fMRI performance uniformly low).

Experimental setup

Datasets: BraTS-MEN BraTS-METS fastMRI VQA-RAD BraTS-2021 ISLES-2022 Br35H RadImageNet RSNA NOVA PubMedVision Baby Open Brains

Benchmarks: OmniBrainBench

Metrics: accuracy ROUGE-1 ROUGE-L BLEU BERTScore

Baselines: Janus-Pro-7B InternVL3 Qwen2.5-VL Qwen3-VL MedVLM-R1 MedGemma LLaVA-Med Lingshu HuatuoGPT-Vision GPT-4o GPT-5 GPT-5-mini

Builds on: VQA-RAD Slake NOVA Brain Tumor VQA

Closest papers by meaning

• OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.91)
• Visual and Domain Knowledge for Professional-level Graph-of-Thought Medical Reasoning (ICML 2025 · similarity 0.90)
• Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks (ICLR 2026 · similarity 0.89)
• Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning (CVPR 2026 · similarity 0.88)
• GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI (NeurIPS 2024 · similarity 0.88)
• M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding (ICLR 2026 · similarity 0.88)