Research map / Time series / Foundation models for time series
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GEM, an MLLM that fuses ECG time series and 12-lead ECG images with text, trained on a 30K GPT-4o-generated grounding dataset (ECG-Grounding) and evaluated on a new Grounded ECG Understanding task.
Problem
Existing language-based ECG models use only one non-text modality (time series or images) and provide little explainability linking diagnoses to granular waveform evidence.
Approach
A dual-encoder MLLM (ECG-CoCa time series encoder plus CLIP image encoder) with MLP projectors into the LLM token space, trained in one step on knowledge-guided instruction data where heartbeat-level feature sequences are fed to GPT-4o via a diagnosis guider to produce grounded answers.
Key results
- On Grounded ECG Understanding (MIMIC-IV-ECG), SFT LLaVA GEM reaches 87.24% diagnosis accuracy vs. PULSE 81.14%; out-domain PTB-XL 73.53-73.59% vs. 59.24%.
- On ECG-Bench CSN classification, GEM (SFT LLaVA) reaches 92.6 accuracy vs. PULSE 85.2; GEM (SFT PULSE) achieves best PTB-XL Super AUC 83.4 vs. 82.4.
- PTB-XL report score rises from 61.3 (PULSE) to 67.1 (GEM SFT PULSE); ECG-QA accuracy is comparable (73.6 vs 73.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune with GEM's multimodal framework | PULSE · MIMIC-IV-ECG | Diagnosis Accuracy: 81.14 -> 86.49 | Helped |
| Fine-tune with GEM's multimodal framework | PULSE · PTB-XL | Diagnosis Accuracy: 59.24 -> 73.59 | Helped |
| Fine-tune with GEM's multimodal framework | PULSE · MIMIC-IV-ECG | ECGFeatureGrounding: 50.18 -> 74.95 | Helped |
| Fine-tune with GEM's multimodal framework | PULSE · PTB-XL Super | AUC: 82.4 -> 83.4 | Helped |
| Fine-tune with GEM's multimodal framework | PULSE · ECG-QA | Accuracy: 73.8 -> 73.6 | No clear effect |
| Add ECG images to time-series input (ablation) | GEM · PTB-XL Super | AUC: 81.2 -> 82.7 | Helped |
| Add ECG images to time-series input (ablation) | GEM · CODE-15% | F1: 84.9 -> 86.3 | Helped |
| Add ECG images to time-series input (ablation) | GEM · CPSC 2018 | F1: 54.0 -> 51.5 | Hurt |
| Add ECG images to time-series input (ablation) | GEM · CSN | ACC: 91.6 -> 90.1 | Hurt |
| Add ECG images to time-series input (ablation) | GEM · ECG-Bench abnormality detection | F1 / ACC: 84.9 -> 86.3; 54.0 -> 51.5; 91.6 -> 90.1 | Mixed |
Surprising findings
- GEM SFT LLaVA, never pretrained on ECG data, performs comparably to or better than SFT PULSE on several ECG-Bench tasks after one epoch.
- Cardiologists rated GEM's findings novelty lower (3.9/5) than GPT-4o (4.4/5) and Deepseek-R1 (4.5/5).
Limitations
- GPT-4o sometimes produces target answers misaligned with cardiologist interpretations (e.g., missed ischemia).
- Failure cases include missed subtle ST-segment or P-wave changes and overstated severity due to lack of patient context.
- Evaluation of grounded understanding relies on GPT-4o scoring, and the model is not intended for standalone clinical use.
Experimental setup
Datasets: ECG-Instruct (PULSE) ECG-Grounding (MIMIC-IV-ECG derived) MIMIC-IV-ECG PTB-XL CPSC2018 G12EC CODE-15% CSN ECG-QA
Benchmarks: Grounded ECG Understanding (MIMIC-IV-ECG, PTB-XL) ECG-Bench (PTB-XL Super, CODE-15%, CPSC 2018, CSN, G12EC, PTB-XL Report, ECG-QA)
Metrics: diagnosis accuracy analysis completeness analysis relevance lead assessment coverage lead assessment accuracy ECG feature grounding evidence-based reasoning clinical diagnostic fidelity AUC F1
Baselines: PULSE GPT-4o ECG-Chat JoLT
Builds on: LLaVA PULSE ECG-CoCa
Closest papers by meaning
- ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretation (ICML 2026 · similarity 0.90)
- ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation (NeurIPS 2026 · similarity 0.89)
- anyECG-chat: A Generalist ECG-MLLM for Flexible ECG Input and Multi-Task Understanding (AAAI 2026 · similarity 0.88)
- From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining (ICML 2025 · similarity 0.88)
- Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model (ICLR 2025 · similarity 0.87)
- Extending Pretrained 10-Second ECG Foundation Models to Longer Horizons (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.