Research map / Vision-language and multimodal / Multimodal LLMs

OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM

CVPR 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 71 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OmniMedVQA, a 127,995-item multi-choice medical VQA benchmark built from 73 medical classification datasets via GPT-3.5-generated questions and distractors, and evaluates 12 general and medical LVLMs with two scoring methods.

Problem

Existing medical VQA datasets are small and cover few modalities and anatomical regions, so LVLMs lack a comprehensive evaluation on real medical images.

Approach

Convert existing medical classification datasets into multi-choice VQA items using classification attributes, with ChatGPT-3.5 used to rephrase questions and generate incorrect options, and evaluate LVLMs with both question-answering and prefix-based (likelihood) scores.

Key results

• BLIP-2 achieves the best overall question-answering accuracy of 50.69% on OmniMedVQA, above all medical LVLMs (best medical: MedVInT 41.50%).
• Most LVLMs only slightly exceed the 28.28% random-guess baseline, with LLaVA-Med at 28.78% (QA score) and 24.06% (prefix score) overall.
• Medical LVLMs perform better on CT, MRI and X-Ray but lag general models on modalities like fundus photography and infrared reflectance imaging.

Measured effects reported in the paper

Surprising findings

• Medical-specialized LVLMs underperform general-domain models overall, with LLaVA-Med the weakest medical model.
• The general-domain BLIP-2 outperforms all medical LVLMs on average, suggesting medical adaptation has not yielded the expected emergent gains.

Limitations

• 31 of 73 source datasets are restricted access, so only QA items are released for them and users must obtain original images.
• Questions derive from classification labels, which are relatively simple and may not test complex clinical reasoning.
• Incorrect options and question rephrasing are GPT-3.5 generated, with only a human double-check for quality control.

Experimental setup

Datasets: VQA-RAD SLAKE Path-VQA VQA-Med RadImageNet Fitzpatrick 17k ISIC2019 Kvasir BreakHis Retinal OCT-C8

Benchmarks: OmniMedVQA

Metrics: accuracy question-answering score prefix-based score

Baselines: MiniGPT-4 BLIP-2 InstructBLIP mPLUG-Owl Otter LLaVA LLaMA-Adapter-v2 VPGTrans Med-Flamingo RadFM MedVInT LLaVA-Med

Builds on: LLaVA InstructBLIP LLaVA-Med MedVInT

Closest papers by meaning

• Falcon Medical Visual Question Answering (AAAI 2025 · similarity 0.91)
• GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI (NeurIPS 2024 · similarity 0.91)
• OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks (CVPR 2026 · similarity 0.91)
• Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine (AAAI 2025 · similarity 0.91)
• MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models (ICLR 2025 · similarity 0.90)
• OpenMedReason: Scientific Reasoning Supervision for Medical Vision–Language Models (NeurIPS 2026 · similarity 0.90)