Research map / Video / Video segmentation and tracking
SleepVST: Sleep Staging from Near-Infrared Video Signals using Pre-Trained Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SleepVST, a transformer pre-trained on contact-sensor cardio-respiratory waveforms (SHHS, MESA) and transferred to near-infrared video-derived waveforms plus motion features for four-class contactless sleep staging.
Problem
Camera-based sleep staging from near-infrared video is limited by small video datasets and by coarse video-derived heart and breathing rate features that lose inter-beat and inter-breath information.
Approach
Pre-train a transformer end-to-end on large contact-sensor cardiac and respiratory waveforms (ECG, thoracic belt), then freeze it as a feature extractor on video-derived waveforms and train a Random Forest on its outputs together with optical-flow motion features.
Key results
- On SHHS and MESA contact-sensor four-class staging, SleepVST reaches total Cohen's kappa 0.75 and 0.77 (accuracy 83.0% and 85.2%).
- On the 50-night OSV video dataset, SleepVST achieves total accuracy 78.8% and Cohen's kappa 0.71, versus 0.64 kappa for the prior video method of Carter et al. (2023).
- Ablation: direct application of the pre-trained model to video gives kappa 0.536, while transfer with a new classifier plus motion features gives 0.708.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Transfer with a newly trained classifier instead of directly applying the pre-trained model (ablation) | SleepVST · Oxford Sleep Volunteers | Cohen’s κT: 0.536 -> 0.689 | Helped |
| Add motion features to the transferred classifier (ablation) | SleepVST · Oxford Sleep Volunteers | Cohen’s κT: 0.689 -> 0.708 | Helped |
| Add motion features during transfer (ablation) | SleepVST · Oxford Sleep Volunteers | Cohen’s κF: 0.397 -> 0.491 | Helped |
| Use motion definitions f1 and f2 rather than no motion features (ablation) | SleepVST · Oxford Sleep Volunteers | Cohen’s κF: 0.397 -> 0.467 | Helped |
| Use motion definitions f3 and f4 rather than no motion features (ablation) | SleepVST · Oxford Sleep Volunteers | Cohen’s κF: 0.397 -> 0.465 | Helped |
| Use all four motion feature definitions rather than only f1 and f2 (ablation) | SleepVST · Oxford Sleep Volunteers | Cohen’s κT: 0.697 -> 0.708 | Helped |
| Use all four motion feature definitions rather than only f3 and f4 (ablation) | SleepVST · Oxford Sleep Volunteers | Cohen’s κT: 0.704 -> 0.708 | Helped |
| Use head, body, and outer-bed regions separately instead of head and body only (ablation) | SleepVST · Oxford Sleep Volunteers | Cohen’s κT: 0.702 -> 0.708 | Helped |
| Use head, body, and outer-bed regions separately instead of combined head/body and outer-bed regions (ablation) | SleepVST · Oxford Sleep Volunteers | Cohen’s κT: 0.704 -> 0.708 | Helped |
| Use a heart waveform instead of heart rate, with breathing rate held fixed (ablation) | SleepVST · SHHS | Cohen’s κT: 0.683 -> 0.718 | Helped |
| Use a breathing waveform instead of breathing rate, with heart rate held fixed (ablation) | SleepVST · MESA | Cohen’s κT: 0.722 -> 0.758 | Helped |
| Use both heart and breathing waveforms instead of both derived rates (ablation) | SleepVST · Oxford Sleep Volunteers | Cohen’s κT: 0.681 -> 0.708 | Helped |
Surprising findings
- Directly applying the contact-sensor pre-trained model to video waveforms performs much worse (kappa 0.536) than training a new classifier on its features, indicating sensor-specific features do not transfer.
Limitations
- Video dataset is small (50 nights) and has limited older-adult representation, with performance decreasing with age.
- Video waveforms depend on the proprietary Oxehealth Vital Signs software for extraction.
Experimental setup
Datasets: SHHS MESA Oxford Sleep Volunteers (OSV)
Benchmarks: SHHS MESA Oxford Sleep Volunteers
Metrics: Cohen's kappa (total) Cohen's kappa (per-night mean) accuracy (total) accuracy (per-night mean) kappa on fragmented sleep (kappa_F)
Baselines: Bakker et al. Sridhar et al. Kotzen et al. (SleepPPG-Net) Carter et al. (2023) van Meulen et al. Kamon et al. Nochino et al.
Builds on: PatchTST ResNet Transformer Pan-Tompkins algorithm Carter et al. video sleep staging
Closest papers by meaning
- Subject-Relative Micro-Motion and Sleep Dynamics for Near-Infrared Video Sleep Staging (NeurIPS 2026 · similarity 0.89)
- Personalized Sleep Staging Leveraging Source-free Unsupervised Domain Adaptation (AAAI 2025 · similarity 0.82)
- SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals (ICML 2024 · similarity 0.82)
- Fusion-Vital: Video-RF Fusion Transformer for Advanced Remote Physiological Measurement (AAAI 2024 · similarity 0.81)
- FacePhys: State of the Heart Learning (NeurIPS 2026 · similarity 0.81)
- Generalizable Sleep Staging via Multi-Level Domain Alignment (AAAI 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.