Research map / Time series / Foundation models for time series
Integrating Sequence and Image Modeling in Irregular Medical Time Series Through Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a joint sequence-and-image framework for irregular medical time series, combining a GAN-based GRU imputer with a Swin Transformer over six time-series-to-image transforms, trained with three self-supervised losses.
Problem
Classifying irregular, heavily missing medical time series (PAM, P12, P19) is hard when methods model only sequences or only image renderings; robustness under additional missingness is also limited.
Approach
Jointly learn a GAN-imputed sequence representation (BiGRU generator, RNN discriminator) and a pretrained Swin Transformer image representation over six time-series-to-image transforms, fused with three SSL losses: inter-sequence NT-Xent, sequence-image contrastive loss with margin, and a clustering loss on joint features.
Key results
- On PAM, the method reaches 98.3% accuracy and 98.5% F1, versus 95.2% accuracy for the second-best ViTST.
- On P19 the method reports 59.6 AUPRC and 91.6 AUROC, and on P12 50.4 AUPRC and 86.0 AUROC, both best among the compared methods.
- Under 50% extra leave-sensors-out missingness on PAM, the method outperforms ViTST by 6.1% accuracy and 4.6% F1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine sequence and image representations (ablation) | Ours · PAM | F1 score: increasing by 2.6% and 4.5% | Helped |
| replace MSE reconstruction loss with NT-Xent (ablation) | sequence encoder · PAM | Accuracy: 0.8% | Helped |
| replace MSE reconstruction loss with NT-Xent (ablation) | sequence encoder · PAM | Precision: 0.6% | Helped |
| replace MSE reconstruction loss with NT-Xent (ablation) | sequence encoder · PAM | Recall: 0.2% | Helped |
| replace MSE reconstruction loss with NT-Xent (ablation) | sequence encoder · PAM | F1 score: 0.5% | Helped |
| add contrastive learning to the joint representation (ablation) | Ours · PAM | Accuracy: 1.1% | Helped |
| add contrastive learning to the joint representation (ablation) | Ours · PAM | F1 score: 1.0% | Helped |
| add clustering loss to the joint representation (ablation) | Ours · PAM | Accuracy: 1.2% | Helped |
| add clustering loss to the joint representation (ablation) | Ours · PAM | F1 score: 0.8% | Helped |
| use the full sequence-image framework and self-supervised strategies rather than the strongest baseline | Ours vs. ViTST · PAM | Accuracy: 3.1% | Helped |
| use the full sequence-image framework and self-supervised strategies rather than the strongest baseline | Ours vs. PrimeNet · P12 | AUROC: 0.9% | Helped |
| use the full sequence-image framework and self-supervised strategies rather than the strongest baseline | Ours vs. PrimeNet · P12 | AUPRC: 1.1% | Helped |
Surprising findings
- Clinical Turing test: five ICU clinicians distinguished real from model-imputed P19 samples at near-chance accuracy (kappa -0.03).
Limitations
- Authors state that data will be released later; data availability is not yet provided.
- Reported gains on P12 and P19 are modest in AUROC and AUPRC relative to some baselines, and results are reported as averages over five runs with some large standard deviations.
- Heavy computational pipeline with six image transforms, GAN imputation and three SSL losses adds complexity (not quantified).
Experimental setup
Datasets: PAM P12 P19
Benchmarks: PAM P12 P19
Metrics: Accuracy Precision Recall F1 score AUROC AUPRC
Baselines: GRU-D SeFT CARD Raindrop PrimeNet ContiFormer ViTST
Builds on: GRU-D ViTST GAN NT-Xent Swin Transformer
Closest papers by meaning
- Sequential Multi-Dimensional Self-Supervised Learning for Clinical Time Series (ICML 2023 · similarity 0.89)
- Time Series as Images: Vision Transformer for Irregularly Sampled Time Series (NeurIPS 2023 · similarity 0.89)
- Autoregressive Sequence Modeling for 3D Medical Image Representation (AAAI 2025 · similarity 0.88)
- Learning Without Augmenting: Unsupervised Time Series Representation Learning via Frame Projections (NeurIPS 2025 · similarity 0.86)
- Revitalizing Medical Time Series with Vision-Informed Retrieval: A Vision-Language Perspective (NeurIPS 2026 · similarity 0.86)
- Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.