Research map / Vision-language and multimodal / Multimodal LLMs

GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and a Comprehensive Multimodal Dataset Towards General Medical AI

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds GMAI-VL-5.5M, a 5.5M-sample medical image-text dataset generated from 219 annotated medical imaging datasets via GPT-4o with annotation-guided prompts, and trains GMAI-VL (InternLM2.5-7B + CLIP encoder + MLP) with a three-stage curriculum that reaches SOTA on medical VQA benchmarks.

Problem

General-purpose LVLMs lack specialized medical knowledge, and existing medical vision-language datasets are limited in modality, data source, or task coverage, with weak adaptation strategies for medical multimodal data.

Approach

Converts structured annotations (modality, department, label, bounding box) from hundreds of specialized medical imaging datasets into image-text pairs and QA via GPT-4o prompts, and trains a LLaVA-style model in three stages (shallow alignment with frozen LLM and encoder, deep alignment unfreezing the encoder, then full instruction tuning).

Key results

• GMAI-VL reaches 88.48% overall on OmniMedVQA, versus 79.96% for its variant trained without GMAI-VL-5.5M data.
• GMAI-VL scores 62.43% on the GMAI-MMBench test set and 51.3% on the MMMU Health & Medicine val track.
• On VQA-RAD, SLAKE and PMC-VQA, GMAI-VL averages 64.5%, while HuatuoGPT-Vision-7B averages 63.7%.

Surprising findings

• Stage 1+3 without deep alignment reaches a higher OmniMedVQA score (86.62) than stage 1+2 (84.64), but the full three-stage combination is needed for the best results.
• GMAI-VL is below HuatuoGPT-Vision-7B on SLAKE (72.9 vs 74.5) and the report notes difficulty distinguishing mild from moderate severity.

Limitations

• Struggles to distinguish subtle severity differences, such as mild versus moderate disease.
• Dataset quality relies on GPT-4o generation; 14 open-source datasets were found to have quality issues.
• Many training sources are proprietary or restricted, and the paper does not clearly release data or weights.

Experimental setup

Datasets: GMAI-VL-5.5M PubMedVision MIMIC-CXR CheXpertPlus PMC-VQA ALLaVA ShareGPT4V LLaVA-Med-60k SA-Med2D-20M

Benchmarks: VQA-RAD SLAKE PMC-VQA OmniMedVQA GMAI-MMBench MMMU Health & Medicine

Metrics: accuracy

Baselines: GPT-4o HuatuoGPT-Vision-7B HuatuoGPT-Vision-34B MedDr InternVL2-40B LLaVA-Med LLaVA-v1.6 Qwen-VL-Chat

Builds on: LLaVA SA-Med2D-20M InternLM2.5

Closest papers by meaning

• GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI (NeurIPS 2024 · similarity 0.94)
• Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation (NeurIPS 2025 · similarity 0.91)
• Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine (AAAI 2025 · similarity 0.90)
• MIMO: A Medical Vision Language Model with Visual Referring Multimodal Input and Pixel Grounding Multimodal Output (CVPR 2025 · similarity 0.90)
• VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge (CVPR 2025 · similarity 0.90)
• OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.89)