Research map / Vision-language and multimodal / Multimodal LLMs

MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosis

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds MedTVT-QA, an ECG/CXR/lab instruction dataset with physiological and disease-level chain-of-evidence QA, and trains MedTVT-R1, a LLaMA3.2-1B-based MLLM with a modality perception layer and GRPO using a Jaccard reward for multi-disease diagnosis.

Problem

Multi-disease diagnosis and interpretable clinical reasoning from heterogeneous multimodal data (ECG time series, chest X-ray images, lab tables) is underexplored; existing MLLMs are single-modality or only give simple presence/absence predictions.

Approach

Combines a cyclic cross-modal attention plus contribution-aware gating (Modality Perception Layer) into an LLM with a three-stage PT/SFT/RFT pipeline, and uses a set-overlap Jaccard reward inside GRPO to reward correct multi-disease sets.

Key results

• MedTVT-R1 reaches F1 0.5190 and recall 0.5908 on disease-level diagnosis, versus 0.1995 F1 for the best baseline (Qwen2.5-VL-3B-Instruct).
• Removing RFT lowers disease-level F1 from 0.5190 to 0.4992; removing pre-training lowers it to 0.4672.
• On physiological-level LAB-QA, MedTVT-R1 gets BLEU 0.1807 and BERTScore 0.8855 versus best baseline BLEU 0.0118.

Surprising findings

• Ablating the ECG modality causes the largest drop in METEOR and ROUGE, which the authors attribute to cardiac relevance of many diseases.
• Baseline MLLMs are compared after converting ECG signals to images and LAB tables to text, since no existing MLLM handles all three modalities.

Limitations

• Relies on a limited set of 8,706 patient combinations from one source (MIMIC-IV) and a small test split of 375 samples.
• Only three modalities (ECG, CXR, LAB) are used; patient history, genomics and other biomarkers are absent.
• Diagnostic QA labels and reasoning are generated by GPT-4o and reviewed by professionals, so they inherit generator biases and may not reflect true clinical reasoning.

Experimental setup

Datasets: MedTVT-QA MIMIC-IV MIMIC-IV-ECG MIMIC-CXR-JPG MIMIC-IV-ECG-EXT-ICD

Benchmarks: MedTVT-QA test split (375 samples)

Metrics: BLEU METEOR ROUGE BERTScore clinical efficacy precision recall F1 AUC

Baselines: InternVL3-1B LLaVA-1.5-7B LLaVA-One-Vision-7B Qwen2.5-VL-3B-Instruct Mini-InternVL-Chat-2B-V1-5 Molmo-7B-O-0924 Deepseek-VL-1.3B-Chat LLaVA-NeXT-8B

Builds on: GRPO DeepSeek-R1 LoRA MuMu-LLaMA Symile

Closest papers by meaning

• MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text (AAAI 2026 · similarity 0.90)
• Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning (CVPR 2026 · similarity 0.90)
• OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning (CVPR 2026 · similarity 0.88)
• RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis (NeurIPS 2025 · similarity 0.88)
• QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training (NeurIPS 2025 · similarity 0.88)
• Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search (NeurIPS 2025 · similarity 0.88)