Research map / Time series / Foundation models for time series
JoLT: Jointly Learned Representations of Language and Time-Series for Clinical Time-Series Interpretation (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JoLT, which jointly learns time-series and text representations via a masked-pretrained time-series transformer encoder, Q-Former alignment, and an OPT decoder, for ECG summarization on PTB-XL.
Problem
Clinical time-series and text are usually modeled in isolation or by converting time-series to images/graphs, leaving time-series summarization and question answering understudied.
Approach
Explicitly models time-series tokens with a masked-reconstruction pretrained transformer encoder, aligned to an OPT language decoder through a Q-Former trained with contrastive, generation, and matching losses, rather than rendering signals as images.
Key results
- JoLT reaches ROUGE-1 F1 0.436 and ROUGE-2 F1 0.295 on PTB-XL, versus 0.363 and 0.172 for fine-tuned BLIP-2.
- JoLT achieves METEOR 0.414 and BLEURT -0.502, outperforming fine-tuned BLIP-2 (0.349, -0.737).
- Zero-shot BLIP and BLIP-2 perform poorly (ROUGE-1 F1 0.007 and 0.044), showing need for domain-specific fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune BLIP-2 on PTB-XL instead of evaluating zero-shot | BLIP-2 · PTB-XL test | ROUGE-1 F1: 0.044 -> 0.363 | Helped |
| Explicitly model time-series with JoLT rather than use fine-tuned image-captioning BLIP-2 | ECG summarization · PTB-XL test | ROUGE-1 F1: 0.363 -> 0.436 | Helped |
| Explicitly model time-series with JoLT rather than use fine-tuned image-captioning BLIP-2 | ECG summarization · PTB-XL test | BLEURT: -0.737 -> -0.502 | Helped |
Surprising findings
- Off-the-shelf zero-shot BLIP and BLIP-2 perform near zero on ECG interpretation, indicating image captioning models do not transfer to clinical waveforms.
Limitations
- Time-series encoder pretrained only on a small subset of PTB-XL data.
- Decoder is not medically pretrained; authors suggest medical-corpus decoders would help.
- Evaluated only on a single ECG dataset and only on summarization.
Experimental setup
Datasets: PTB-XL
Benchmarks: PTB-XL
Metrics: ROUGE-1 ROUGE-2 ROUGE-L METEOR BLEURT
Baselines: BLIP BLIP-2 fine-tuned BLIP-2
Builds on: Q-Former BLIP-2 OPT
Closest papers by meaning
- ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Dataset (ICML 2025 · similarity 0.85)
- A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle (ICML 2025 · similarity 0.85)
- Revitalizing Medical Time Series with Vision-Informed Retrieval: A Vision-Language Perspective (NeurIPS 2026 · similarity 0.85)
- TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting (ICLR 2024 · similarity 0.85)
- TRACE: Grounding Time Series in Context for Multimodal Embedding and Retrieval (NeurIPS 2025 · similarity 0.85)
- MIRA: Medical Time Series Foundation Model for Real-World Health Data (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.