Research map / Vision-language and multimodal / Multimodal LLMs

ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ECG-Reasoning-Benchmark, a multi-turn, pipeline-grounded benchmark (~6,400 cases, 17 diagnoses) that step-by-step verifies MLLM ECG reasoning, finding that current models reach under 6% full-chain completion despite high initial diagnosis accuracy.

Problem

Existing ECG-MLLM evaluations rely on LLM-as-a-Judge against synthetic references, so it is unclear whether models perform genuine step-by-step, signal-grounded clinical reasoning or rely on superficial visual cues.

Approach

Derive verifiable ground truth reasoning chains from an automated ECG analysis pipeline (U-Net3+ wave delineation plus rule-based findings and diagnosis logic diagrams), then probe models with a 4-step multi-turn loop (criterion selection, finding identification, lead/wave/measurement grounding, diagnostic decision) instead of LLM-judged free text.

Key results

• Completion stays below about 6% for all models, with Gemini-3-Flash highest at 6.26% on PTB-XL and ECG-R1-RL at 5.81% on MIMIC-IV-ECG.
• Average Depth rarely exceeds 2.0 (out of 4), with Gemini-3-Flash at 2.09 on PTB-XL, indicating failures at finding identification and grounding.
• ECG-specific models drop sharply under ground-truth reasoning guidance, e.g. ECG-R1-RL falls from 85.41% IDA to 22.70% GT-RDA on PTB-XL, while Hulu-Med (32B) rises from 57.49% to 99.42%.

Surprising findings

• ECG-specific models (PULSE, ECG-R1, GEM) with high initial diagnosis accuracy collapse when given correct reasoning steps, suggesting pattern matching over genuine deduction.
• Llama-3.2-Vision (11B) underperforms smaller Qwen3-VL (8B), which the authors attribute to lack of ECG/medical training exposure.

Limitations

• Excludes borderline or ambiguous ECGs, so diagnostic uncertainty is not evaluated.
• Requires full traversal of all findings, penalizing clinician-style heuristic shortcuts.
• Ground truth depends on the automated pipeline's rule-based findings, and P-wave precision is affected by LUDB annotation policy.

Experimental setup

Datasets: PTB-XL MIMIC-IV-ECG LUDB

Benchmarks: ECG-Reasoning-Benchmark

Metrics: Initial Diagnosis Accuracy Completion Depth GT-Reasoning-Based Diagnosis Accuracy

Baselines: PULSE GEM ECG-R1 OpenTSLM Hulu-Med MedGemma Qwen3-VL Llama-3.2-Vision Gemini-2.5-Flash Gemini-2.5-Pro Gemini-3-Flash GPT-5-Mini

Builds on: CXReasonBench U-Net3+ LLM-as-a-Judge

Closest papers by meaning

• ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretation (ICML 2026 · similarity 0.92)
• GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images (NeurIPS 2025 · similarity 0.89)
• anyECG-chat: A Generalist ECG-MLLM for Flexible ECG Input and Multi-Task Understanding (AAAI 2026 · similarity 0.88)
• CardioLens: Revealing the Clinical Reality Gap of MLLMs via Multi-Sequence Cardiac MRI Evaluations (NeurIPS 2026 · similarity 0.86)
• Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model (ICLR 2025 · similarity 0.86)
• From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining (ICML 2025 · similarity 0.86)