Research map / Vision-language and multimodal / Multimodal LLMs

MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding

ICML 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MedXpertQA, a 4,460-question expert-level medical benchmark with a text subset and a multimodal subset with clinical-scenario images, and evaluates 18 LLMs/LMMs.

Problem

Existing medical QA and multimodal VQA benchmarks are too easy, lack specialty breadth and clinical detail, and rely on auto-generated QA from captions, so they cannot assess expert-level medical reasoning.

Approach

Builds a difficulty-filtered, specialty-covering medical exam benchmark using AI-expert and human-Brier-score filtering, LLM-based question/option augmentation to reduce leakage, expert review, and a Reasoning vs Understanding split for evaluating o1-like models.

Key results

• GPT-4o reaches 30.37% average on MedXpertQA-Text and 42.80% on MedXpertQA-MM, versus 45.20% for pre-licensed experts on the Understanding set and 43.92% overall on Text.
• o1 scores 44.67% on MedXpertQA-Text (on a 10% sample) and 56.28% on MM, the best among evaluated models, while the best open model Qwen2.5-VL-72B reaches 23.35% on Text average.
• On Text, DeepSeek-R1 reaches 37.76% average versus 24.16% for DeepSeek-V3, and gaps between Reasoning and Understanding narrow for o1-like models.

Measured effects reported in the paper

Surprising findings

• Vanilla LLMs and LMMs score lower on the Reasoning subset than on the Understanding subset, but this gap narrows or reverses for o1-like inference-time scaled models.
• Expert pre-licensed baseline (43.92%) is only modestly above strong models such as o1 on the text subset, showing limited headroom for frontier models.

Limitations

• Full test data is withheld to prevent leakage, so evaluation depends on the authors' release policy.
• o1 and o3-mini were evaluated only on 10% samples due to cost, adding variance.
• Benchmark is multiple-choice and mostly US-centric; authors note performance does not imply clinical readiness.

Experimental setup

Datasets: USMLE COMLEX-USA ACR DXIT/TXIT EDiR NEJM Image Challenge ABMS specialty board exams

Benchmarks: MedXpertQA-Text MedXpertQA-MM

Metrics: accuracy

Baselines: GPT-4o o1 DeepSeek-R1 Claude-3.5-Sonnet Gemini-1.5-Pro Gemini-2.0-Flash Qwen2.5-VL-72B Qwen2-VL-72B QVQ-72B-Preview o3-mini

Builds on: MedQA MMMU OmniMedVQA MMLU-Pro

Closest papers by meaning

• MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text (AAAI 2026 · similarity 0.91)
• Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning (CVPR 2026 · similarity 0.89)
• Medical thinking with multiple images (ICLR 2026 · similarity 0.89)
• MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.88)
• Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.88)
• MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models (ICLR 2025 · similarity 0.88)