Research map / Video / Video generation
ECHOPulse: ECG Controlled Echocardio-gram Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ECHOPulse, a VQ-VAE tokenizer plus ECG-conditioned masked visual token transformer that generates echocardiogram videos from ECG signals and an optional conditioning image.
Problem
Echocardiogram video generation for synthetic data and monitoring is slow with diffusion models and relies on expert-curated conditions (masks, text, optical flow) that are costly to obtain; ECG is readily available but unused for ECHO video synthesis.
Approach
Discretize echo videos with a VQ-VAE/LFQ video tokenizer (domain-transferred from natural video), and align ECG patch embeddings from an ECG foundation model with video tokens in a bidirectional masked token transformer (MaskGIT-style), enabling fast parallel decoding and ECG-only conditioning with autoregressive extension to long videos.
Key results
- On the private dataset, ECG-conditioned ECHOPulse with domain transfer reaches FID 15.50 (A2C) and 20.82 (A4C), FVD 82.44/107.40, and SSIM 0.67/0.66, the best among compared settings.
- LVEF estimation from generated videos gives MAE 2.51 and RMSE 2.86 vs EchoDiffusion's 4.81 and 6.69, though R^2 is lower (0.85 vs 0.89).
- Generation of 64 frames takes 6.4 s vs 146 s for EchoDiffusion, with 279M vs 381M parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the video tokenizer on ECHO data (domain transfer) (ablation) | ECHOPulse · Echo-Dynamic reconstruction | MSE: 5.02 \times 10^{-3} -> 2.89 \times 10^{-3} | Helped |
| Fine-tune the video tokenizer on ECHO data (domain transfer) (ablation) | ECHOPulse · Private data reconstruction | MSE: 6.09 \times 10^{-3} -> 2.91 \times 10^{-3} | Helped |
| Use ECG conditioning instead of text conditioning | ECHOPulse (Domain transfer) · Private data, A2C | FID: 25.44 -> 15.50 | Helped |
| Use ECG conditioning instead of text conditioning | ECHOPulse (Domain transfer) · Private data, A4C | FID: 31.21 -> 20.82 | Helped |
| Use ECG conditioning instead of text conditioning | ECHOPulse (Domain transfer) · Private data, A2C | SSIM: 0.54 -> 0.67 | Helped |
| Use ECG conditioning instead of text conditioning | ECHOPulse (Domain transfer) · Private data, A4C | SSIM: 0.54 -> 0.66 | Helped |
| Fine-tune on ECHO data rather than use only natural videos (ablation) | ECHOPulse · CAMUS, Text, A2C | FID: 12.71 -> 5.65 | Helped |
| Fine-tune on ECHO data rather than use only natural videos (ablation) | ECHOPulse · CAMUS, Text, A4C | FID: 15.38 -> 8.17 | Helped |
| Condition on ECG with an image rather than image plus EF | ECHOPulse · LVEF evaluation | MAE: 4.81 -> 2.51 | Helped |
| Condition on ECG with an image rather than image plus EF | ECHOPulse · LVEF evaluation | R²: 0.89 -> 0.85 | Hurt |
Surprising findings
- Reconstruction after only one epoch of domain transfer from natural videos surpasses the diffusion-based EchoNet-Synthetic on MSE/MAE, despite the VQ-VAE starting worse.
- The model generalizes zero-shot to ECG recorded from an Apple Watch.
Limitations
- Private dataset cannot be released due to privacy concerns; anonymization cannot be automatically guaranteed.
- Fixed low resolution (128x128) and frame rate; lacks FPS and resolution improvements.
- Text-conditioned variant lags on FVD and some baselines (HeartBeat, EchoNet-Synthetic with richer conditions) remain better on certain metrics.
Experimental setup
Datasets: WebVid-10M CAMUS EchoNet-Dynamic private ECHO-ECG dataset (94,078 videos)
Benchmarks: CAMUS EchoNet-Dynamic private ECHO dataset
Metrics: FID FVD SSIM MSE MAE LVEF R^2 LVEF MAE LVEF RMSE sampling time
Baselines: EchoNet-Synthetic HeartBeat EchoDiffusion MoonShot VideoComposer
Builds on: VQ-VAE MAGVIT2 MaskGIT C-ViViT ST-MEM
Closest papers by meaning
- EchoXFlow: A Beamspace Echocardiography Dataset for Cardiac Motion, Flow, and Function (NeurIPS 2026 · similarity 0.86)
- Leveraging an ECG Beat Diffusion Model for Morphological Reconstruction from Indirect Signals (NeurIPS 2024 · similarity 0.84)
- Self-supervised Learning of Echocardiographic Video Representations via Online Cluster Distillation (NeurIPS 2025 · similarity 0.84)
- From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining (ICML 2025 · similarity 0.83)
- Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model (ICLR 2025 · similarity 0.83)
- CardioLens: Revealing the Clinical Reality Gap of MLLMs via Multi-Sequence Cardiac MRI Evaluations (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.