Research map / Vision-language and multimodal / Multimodal LLMs

CardioLens: Revealing the Clinical Reality Gap of MLLMs via Multi-Sequence Cardiac MRI Evaluations

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CardioLens, a private multi-sequence cardiac MRI benchmark (473,896 slices, 13,494 QA pairs) that evaluates 24 MLLMs on image understanding, report generation, and diagnosis, finding a steep performance decline along the clinical workflow and category collapse.

Problem

Existing medical MLLM benchmarks rely on leakage-prone public data, isolated images, and simplified recognition-style tasks, so they do not measure clinically grounded interpretation of multi-sequence cardiac MRI studies.

Approach

A leakage-resistant, report-derived CMR benchmark organized into three workflow tiers (per-sequence image understanding, sequence-level report generation, patient-level diagnosis), with controlled analyses of slice-selection protocols, slice budget, and reasoning prompts to test whether input construction explains the gap.

Key results

• Across 24 MLLMs, performance declines along the workflow; QoQMed-7B drops from 58.8% F1 on image understanding to 49.7% on report generation and 20.5% on diagnosis.
• Slice-selection strategies (random, heuristic, MIL) change F1 by typically under 2% for most models; increasing slice budget from 8 to 128 changes F1 by at most 1.3% for the tested models.
• Reasoning prompts reduce recall, e.g., MedGemma-4B recall falls from 67.44% to 53.77%, with mostly negative effects on F1 for most models.

Surprising findings

• Gemma-4-31B predicts NICM for 88% of IHD cases and 53% of normal hearts, showing category collapse despite 50.8% accuracy.
• Step-by-step reasoning prompts generally lower recall and F1 rather than improving them.

Limitations

• Only 100 of 585 patients are publicly released because of data-sharing constraints.
• Closed-source models were evaluated only on the 100-patient public subset, not the full cohort.

Experimental setup

Datasets: private multi-sequence CMR archive (585 patients, 5,969 files)

Benchmarks: CardioLens

Metrics: accuracy precision recall F1

Baselines: GPT-5 Gemini-3.1-Pro Claude-Opus-4.6 MedGemma Lingshu

Builds on: MIL

Closest papers by meaning

• CMR-RD: Long-Tailed Adaptive VLM for Explainable CMR Diagnosis (CVPR 2026 · similarity 0.91)
• OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks (CVPR 2026 · similarity 0.87)
• How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.87)
• MedHorizon: Towards Long-context Medical Video Understanding in the Wild (NeurIPS 2026 · similarity 0.87)
• MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models (ICLR 2025 · similarity 0.86)
• ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation (NeurIPS 2026 · similarity 0.86)