Research map / Vision-language and multimodal / Multimodal LLMs

JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces JMed48k, a 48,862-question Japanese multi-profession healthcare licensing benchmark with images, and a paired image-removal audit that classifies answer transitions to test visual evidence use in 21 VLMs.

Problem

Existing medical exam benchmarks are text-based or cover one profession, and aggregate accuracy on image subsets hides whether models actually use images. The paper targets profession-stratified evaluation of vision-language models on licensing exams with item-level image-use measurement.

Approach

Builds a multi-profession Japanese licensing benchmark from official MHLW PDFs with an 8-type visual taxonomy, and evaluates each image-bearing question with and without its images, decomposing outcomes into four answer-transition states (p11, p10, p01, p00) to measure net image effect.

Key results

• Proprietary models gain substantially from images: GPT-5 mini shows net image effect of +23.1 points, while medical-specific models show net effect near zero across the cohort.
• For proprietary models, net image effect ranges from +5.7 on Physician to +39.8 on Public Health Nurse, a sevenfold spread across professions.
• Gemini 2.5 Pro leads text-only macro accuracy at 95.5%, while the strongest medical-specific model Lingshu 32B reaches 65.0%.

Surprising findings

• Medical-specific models score at random on items where answer choices are embedded in images, suggesting their paired-audit gains reflect text priors rather than visual correction.
• Hulu-Med 14B is the only model where image input is net harmful (p01 9.7% exceeds p10 8.7%).

Limitations

• Benchmark measures licensing exam performance rather than clinical deployment.
• The audit measures observable answer changes rather than human-like reasoning.
• Scored subset excludes 93 items without a usable official gold answer and covers only the most recent five years.

Experimental setup

Datasets: JMed48k

Benchmarks: JMed48k-Eval

Metrics: accuracy net image effect p10 p01

Baselines: Gemini 2.5 Flash Gemini 2.5 Pro GPT-5 GPT-5 mini Claude Sonnet 4 Grok 4.20 Qwen3.5 VL 397B-A17B Gemma 4 IT 31B Llama 4 Maverick GLM-4.6V 9B DeepSeek-R1 Lingshu 32B

Builds on: MinerU MedQA KokushiMD-10

Closest papers by meaning

• CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models (NeurIPS 2024 · similarity 0.87)
• OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.87)
• Medical thinking with multiple images (ICLR 2026 · similarity 0.87)
• Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs (AAAI 2025 · similarity 0.87)
• MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence (NeurIPS 2026 · similarity 0.87)
• OpenMedReason: Scientific Reasoning Supervision for Medical Vision–Language Models (NeurIPS 2026 · similarity 0.86)