Research map / Vision-language and multimodal / Human and embodied modeling
SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds SleepFM, a multi-modal contrastive foundation model pretrained on ~100k hours of Stanford polysomnography (BAS, ECG, respiratory), and introduces leave-one-out contrastive learning that beats pairwise CL and a supervised CNN on sleep staging, SDB detection, and demographics.
Problem
Sleep analysis from polysomnography typically uses supervised models on narrow tasks and single signals, leaving unlabeled multi-modal physiological structure across brain, cardiac, and respiratory channels unused.
Approach
A leave-one-out contrastive objective in which each modality's embedding is contrasted against the average of the other modalities' embeddings, encouraging a representation integrating all modalities, compared against standard pairwise contrastive learning across three 1D EfficientNet-style encoders.
Key results
- Leave-one-out logistic regression on SleepFM embeddings reaches macro AUROC 0.906 and AUPRC 0.685 for sleep staging vs 0.842 and 0.579 for the supervised CNN.
- For SDB detection, leave-one-out CL gets AUROC 0.941 and AUPRC 0.711 vs 0.843 and 0.555 for the supervised CNN.
- Cross-modal retrieval over 90,000 clips reaches Recall@10 of up to 0.82 with pairwise CL, versus a random baseline of 0.0001.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use leave-one-out CL instead of pairwise CL (ablation) | SleepFM embeddings for sleep stage classification · Sleep stage classification | AUROC: 0.876 -> 0.906 | Helped |
| Use leave-one-out CL instead of pairwise CL (ablation) | SleepFM embeddings for sleep stage classification · Sleep stage classification | AUPRC: 0.608 -> 0.685 | Helped |
| Use leave-one-out CL instead of pairwise CL (ablation) | SleepFM embeddings for SDB classification · SDB classification | AUROC: 0.902 -> 0.941 | Helped |
| Use leave-one-out CL instead of pairwise CL (ablation) | SleepFM embeddings for SDB classification · SDB classification | AUPRC: 0.586 -> 0.711 | Helped |
| Use leave-one-out CL instead of pairwise CL (ablation) | SleepFM embeddings for age classification · Age classification | AUROC: 0.851 -> 0.883 | Helped |
| Use leave-one-out CL instead of pairwise CL (ablation) | SleepFM embeddings for gender classification · Gender classification | AUROC: 0.810 -> 0.850 | Helped |
| Use pairwise CL instead of leave-one-out CL (ablation) | SleepFM cross-modality retrieval · Test set retrieval | Recall@10: 0.58 -> 0.74 | Helped |
| Pretrain with three modalities rather than a single modality (ablation) | SleepFM representations · Sleep stage classification and SDB detection | AUPRC | Helped |
| Include ECG in multimodal pretraining rather than using BAS and respiratory only (ablation) | SleepFM representations · Sleep stage classification and SDB detection | AUPRC | Helped |
| Pretrain with one modality only instead of multiple modalities (ablation) | SleepFM representations · Downstream classification tasks | Performance | Hurt |
| Use BAS-Respiratory pairing instead of modality combinations incorporating ECG (ablation) | SleepFM representations · Downstream classification tasks | Performance | Hurt |
| Use pairwise CL instead of leave-one-out CL (ablation) | SleepFM embeddings for cross-modality retrieval · Test set retrieval | Recall@10 | Helped |
Surprising findings
- Pairwise contrastive learning gave better cross-modality retrieval than leave-one-out, though leave-one-out was better for downstream classification.
- Embeddings from individual modalities (e.g., ECG) carried reasonable sleep-stage performance, and the supervised CNN trained on BAS alone far outperformed multi-modal CNN in some views.
Limitations
- Trained and evaluated primarily on data from a single institution (Stanford sleep clinic).
- Only sleep stage and SDB detection evaluated on downstream tasks; other events like arousal or narcolepsy not studied.
- Baseline CNN trained end-to-end on a much larger dataset (11,261 participants) than the SleepFM probes (1,265), making comparison asymmetric.
Experimental setup
Datasets: Stanford Sleep Clinic polysomnography (14,068 recordings, 1999-2020) Physionet Computing in Cardiology 2018 challenge
Benchmarks: Stanford internal PSG test set Physionet CinC 2018 (external validation)
Metrics: AUROC AUPRC F1 median rank Recall@10
Baselines: Supervised CNN (end-to-end 1D EfficientNet) Pairwise contrastive learning
Builds on: SimCLR CLIP EfficientNet MobileNetV2
Closest papers by meaning
- Robust Sleep Staging over Incomplete Multimodal Physiological Signals via Contrastive Imagination (NeurIPS 2024 · similarity 0.90)
- Subject-Relative Micro-Motion and Sleep Dynamics for Near-Infrared Video Sleep Staging (NeurIPS 2026 · similarity 0.86)
- Generalizable Sleep Staging via Multi-Level Domain Alignment (AAAI 2024 · similarity 0.85)
- What do EEG Foundation Models Capture from Human Brain Signals? (NeurIPS 2026 · similarity 0.85)
- Personalized Sleep Staging Leveraging Source-free Unsupervised Domain Adaptation (AAAI 2025 · similarity 0.85)
- Standing on the Shoulders of Giants: Rethinking EEG Foundation Model Pretraining via Multi-Teacher Distillation (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.