Research map / Vision-language and multimodal / CLIP and zero-shot
SensorLM: Learning the Language of Wearable Sensors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SensorLM, a family of sensor-language foundation models pretrained on 59M+ hours of wearable data with hierarchical (statistical, structural, semantic) auto-generated captions, combining contrastive and captioning objectives.
Problem
Wearable sensor time series are long and high-dimensional, so LLMs cannot process them well, and large-scale paired sensor-text data is lacking for aligning sensor signals with language.
Approach
A hierarchical caption pipeline (statistical, structural, semantic) auto-generates text for minutely wearable features from unlabeled-by-text data, and a unified CLIP/Cap/CoCa-style encoder-decoder is pretrained on these pairs for zero-shot classification, retrieval, few-shot transfer and captioning.
Key results
- Zero-shot 20-class activity recognition: SensorLM AUROC 0.84 vs 0.50-0.51 for Gemma-3-27B and Gemini 2.0.
- Zero-shot sensor-to-text retrieval on 100 samples: R@1 100.0 vs 1.0 (Gemma-3-27B) and 5.0 (Gemini 2.0); at 5,000 samples R@1 98.2.
- Few-shot AR reaches AUROC 0.88 with 50 labels per class, outperforming SimCLR, DINO and MSN baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SensorLM-B rather than the strongest listed LLM baseline | 20-class activity recognition · 20-class activity recognition | AUROC: 0.51 -> 0.84 | Helped |
| Use SensorLM-B rather than the strongest listed LLM baseline | Activity by environmental context · Activity by environmental context | AUROC: 0.51 -> 0.64 | Helped |
| Use SensorLM-B rather than the strongest listed LLM baseline | Cardio vs. strength training · Cardio vs. strength training | AUROC: 0.53 -> 0.71 | Helped |
| Use SensorLM-B rather than the strongest listed LLM baseline | Locomotion vs. stationary · Locomotion vs. stationary | AUROC: 0.55 -> 0.61 | Helped |
| Use SensorLM-B rather than the strongest listed LLM baseline | Fine-grained gym cardio recognition · Fine-grained recognition (gym cardio) | AUROC: 0.52 -> 0.76 | Helped |
| Use SensorLM-B rather than the strongest listed LLM baseline | Fine-grained outdoor sports recognition · Fine-grained recognition (outdoor sports) | AUROC: 0.54 -> 0.83 | Helped |
| Use SensorLM-B rather than Gemini 2.0 | Sensor-to-text retrieval · Sensor-to-text retrieval, 100 samples | Recall@1: 5.0 -> 100.0 | Helped |
| Use structural and semantic captions instead of semantic captions alone (ablation) | SensorLM-B · Activity zero-shot classification | AUROC: 0.71 -> 0.84 | Helped |
| Add statistical captions to semantic captions (ablation) | SensorLM-B · Activity zero-shot classification | AUROC: 0.71 -> 0.66 | Hurt |
| Use statistical captions only instead of structural and semantic captions (ablation) | SensorLM-B · Activity zero-shot classification | AUROC: 0.84 -> 0.51 | Hurt |
| Use structural captions only instead of structural and semantic captions (ablation) | SensorLM-B · Activity zero-shot classification | AUROC: 0.84 -> 0.50 | Hurt |
| Add statistical captions to structural and semantic captions (ablation) | SensorLM-B · Activity zero-shot classification | AUROC: 0.84 -> 0.66 | Hurt |
Surprising findings
- Statistical captions reduce zero-shot activity recognition when added to semantic captions, while they help anxiety and hypertension prediction.
- The Cap-only variant performs near random in zero-shot classification (AUROC 0.55) because it lacks a trained projection head.
Limitations
- Not clinically validated and not intended for medical decision-making.
- Evaluated only on specific wrist-worn Fitbit/Pixel Watch data and a limited feature set.
- LLM baselines were given subsampled tabular data, which may understate their capability.
Experimental setup
Datasets: Fitbit/Pixel Watch wearable sensor data (pretraining) Activity dataset (22,289 person-days, 10,013 individuals) Metabolic dataset (241,532 examples, 1,979 individuals) Sensor-text retrieval dataset (39,766 examples, 975 individuals)
Benchmarks: Activity recognition zero-shot (20-class and concept tasks) Sensor-text cross-modal retrieval Few-shot Activity and Metabolic (Hypertension, Anxiety) tasks Sensor captioning
Metrics: AUROC Macro F1 Balanced Accuracy Recall@1 Recall@5
Baselines: Gemini 2.0 Flash Gemma-3-27B SimCLR DINO MSN
Builds on: CLIP CoCa SimVLM Vision Transformer
Closest papers by meaning
- Limitations in Employing Natural Language Supervision for Sensor-Based Human Activity Recognition - And Ways to Overcome Them (AAAI 2025 · similarity 0.90)
- Scaling Wearable Foundation Models (ICLR 2025 · similarity 0.86)
- HoloLLM: Multisensory Foundation Model for Language-Grounded Human Sensing and Reasoning (NeurIPS 2025 · similarity 0.85)
- ZeroHAR: Sensor Context Augments Zero-Shot Wearable Action Recognition (AAAI 2025 · similarity 0.85)
- EgoLM: Multi-Modal Language Model of Egocentric Motions (CVPR 2025 · similarity 0.85)
- Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.