Research map / Vision-language and multimodal / Cross-modal learning

Limitations in Employing Natural Language Supervision for Sensor-Based Human Activity Recognition - And Ways to Overcome Them

AAAI 2025 · Vision-language and multimodal › Cross-modal learning · approach: Unimodal-to-multimodal representation transfer · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that CLIP-style natural language supervision for wearable-sensor HAR performs far worse zero-shot than supervised and self-supervised baselines, and proposes adapting projection heads on small target data, LLM-diversified text, and ChatGPT-derived external knowledge to close much of the gap.

Problem

Natural language supervision (contrastive sensor-text pre-training) promises plug-and-play zero-shot human activity recognition from wearable accelerometers, but it is unclear whether it works for wearable sensors, where sensor heterogeneity and sparse activity text limit it.

Approach

Identify sensor heterogeneity and lack of rich activity text as the causes of poor NLS performance, then mitigate them by adapting sensor/text projection heads on small target-labeled data, augmenting activity sentences with multiple templates and ChatGPT rewrites, and injecting ChatGPT-generated body-part and movement descriptions.

Key results

• Zero-shot NLS pre-trained on Capture-24 performs far below baselines, e.g., HHAR 31.05 vs 52-59 F1 for supervised/self-supervised baselines, and PAMAP2 10.88 vs ~58-60.
• Adapting projection layers on target data raises Capture-24-pretrained NLS, e.g., HHAR from 31.05 to 58.63 and PAMAP2 from 10.88 to 54.80; combining improved sentences and SLIP objective with CLIP text encoder gives HHAR 50.2 and Mobiact 63.69.
• Adding ChatGPT body-part information to activity sentences improves Mobiact, Motionsense and PAMAP2 results, e.g., PAMAP2 rises to 51.90 in the best template setting, and is overall the best option for unseen-activity recognition.

Measured effects reported in the paper

Surprising findings

• Simple convolutional IMU encoders outperform more complex ResNets during NLS pre-training.
• Zero-shot NLS on Capture-24 is drastically worse (around 30-40%) than simple end-to-end and self-supervised training across six target datasets.

Limitations

• NLS still does not outperform state-of-the-art supervised and self-supervised training on some datasets.
• Adaptation on target labels violates the strict zero-shot setting and requires some labeled target data.
• Zero-shot results are not directly comparable to standard HAR since similarity is computed only among unseen classes.

Experimental setup

Datasets: Capture-24 HHAR Myogym Mobiact Motionsense MHEALTH PAMAP2 RealWorld

Benchmarks: HHAR Myogym Mobiact Motionsense MHEALTH PAMAP2 RealWorld

Metrics: macro F1 top-5 retrieval accuracy (qualitative)

Baselines: Conv. classifier DeepConvLSTM Autoencoder + MLP classifier SimCLR + MLP classifier Enhanced CPC + MLP classifier X-CLIP

Builds on: CLIP SLIP UniCL SimCLR DeepConvLSTM

Closest papers by meaning

• SensorLM: Learning the Language of Wearable Sensors (NeurIPS 2025 · similarity 0.90)
• MoPFormer: Motion-Primitive Transformer for Wearable-Sensor Activity Recognition (NeurIPS 2025 · similarity 0.87)
• RelCon: Relative Contrastive Learning for a Motion Foundation Model for Wearable Data (ICLR 2025 · similarity 0.85)
• Domain-Informed Label Fusion Surpasses LLMs in Free-Living Activity Classification (Student Abstract) (AAAI 2025 · similarity 0.84)
• Spatial-Related Sensors Matters: 3D Human Motion Reconstruction Assisted with Textual Semantics (AAAI 2024 · similarity 0.84)
• Decomposing and Fusing Intra- and Inter-Sensor Spatio-Temporal Signal for Multi-Sensor Wearable Human Activity Recognition (AAAI 2025 · similarity 0.84)