Research map / Vision-language and multimodal / CLIP and zero-shot
From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MELP, an ECG-language pretraining model that aligns ECG signals with clinical reports at token (captioning), beat (beat-sentence contrastive), and rhythm (global contrastive) levels, after cardiology-specific text pretraining.
Problem
Existing ECG-language pretraining methods align ECG with text only globally and miss the multi-scale hierarchical structure of ECG signals (token, beat, rhythm), limiting generalization across downstream tasks.
Approach
Combines three cross-modal supervision levels: an autoregressive ECG captioning loss on token embeddings (token view), an attention-based beat-sentence contrastive loss (beat view), and a global InfoNCE ECG-report loss (rhythm view), with the text encoder initialized from a MedCPT query encoder further pretrained on cardiology text with MLM.
Key results
- Zero-shot classification average AUROC 79.0% vs. MERL 75.3% across six settings (+3.7 average gain).
- Linear probing with 1% training data on PTBXL-Rhythm reaches 88.83% AUROC, versus 81.45% for the second-best ECGFM.
- ECG report generation on 500 PTB-XL samples: BLEU-4 1.87 and BERTScore F1 44.08 vs. PULSE (7B) 0.83 and 10.96.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MELP instead of MERL for zero-shot classification | ECG-language model · Zero-shot ECG classification (Average) | AUC [%]: +3.7% | Helped |
| add token-level language modeling loss to rhythm- and beat-level losses (ablation) | MELP · Linear probing (Average across six tasks) | AUC [%]: 82.32 -> 85.78 | Helped |
| add beat-level local alignment loss to rhythm- and token-level losses (ablation) | MELP · Linear probing (Average across six tasks) | AUC [%]: 82.92 -> 85.78 | Helped |
| add rhythm-level global contrastive loss (ablation) | MELP · Linear probing (Average across six tasks) | AUC [%]: 76.10 -> 85.78 | Helped |
| use only rhythm-level global contrastive loss instead of all three losses (ablation) | MELP · Linear probing (Average across six tasks) | AUC [%]: 85.78 -> 82.51 | Hurt |
| replace Wav2Vec 2.0 with ResNet-18 as ECG encoder (ablation) | MELP · Linear probing (Average across six tasks) | AUC [%]: 85.78 -> 83.07 | Hurt |
| add CMSC pretraining to the Wav2Vec 2.0 ECG encoder (ablation) | MELP · Linear probing (Average across six tasks) | AUC [%]: 85.78 -> 82.97 | Hurt |
| add cardiology-specific text pretraining (ablation) | MELP · Linear probing (Average across six tasks) | AUC [%]: 85.43 -> 85.78 | Helped |
| use 16 learnable beats instead of 10 (ablation) | MELP · Linear probing (Average across six tasks) | AUC [%]: +0.66% | Helped |
| set λ1=2 and λ2=0.2 rather than λ1=1 and λ2=1 (ablation) | MELP · Zero-shot ECG classification (Average) | AUC [%]: 77.1 -> 78.9 | Helped |
Surprising findings
- Removing the rhythm-level global contrastive loss causes the largest drop, with variants lacking it performing markedly worse, while the token-level captioning loss alone gives lower linear-probing averages than the global loss alone.
Limitations
- Token-level supervision lacks explicit clinical interpretability since reports do not describe waveform components explicitly.
- Text encoder is MedCPT rather than a modern LLM, limiting use of LLM advances.
- Report generation evaluation is preliminary and MELP is primarily designed for global representations.
Experimental setup
Datasets: MIMIC-IV-ECG PubMed (cardiology subset) Wikipedia (Cardiology category) PTB-XL CPSC2018 CSN ECGBench
Benchmarks: PTB-XL CPSC2018 CSN ECGBench
Metrics: AUROC BLEU-1 BLEU-4 METEOR ROUGE-L BERTScore F1 Recall@k
Baselines: SimCLR BYOL BarlowTwins MoCo-v3 SimSiam TS-TCC CLOCS ECGFM ST-MEM HeartLang MERL Wav2Vec 2.0 + CMSC + RLM
Builds on: MERL CLIP MedCPT Wav2Vec 2.0 CoCa
Closest papers by meaning
- Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model (ICLR 2025 · similarity 0.92)
- anyECG-chat: A Generalist ECG-MLLM for Flexible ECG Input and Multi-Task Understanding (AAAI 2026 · similarity 0.89)
- Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement (ICML 2024 · similarity 0.88)
- GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images (NeurIPS 2025 · similarity 0.88)
- Parallel-Learning of Invariant and Tempo-variant Attributes of Single-Lead Cardiac Signals: PLITA (AAAI 2025 · similarity 0.88)
- Extending Pretrained 10-Second ECG Foundation Models to Longer Horizons (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.