Research map / Video / Action recognition
MotorSense: A Video-EMG Dataset of Motoric Representations for Action Understanding
Eadom T Dessalene, Michael Maynord, Amir Hossein Shahidzadeh, Botao He, Yasamin Tabatabaee, Ruoshi Liu, Ruohan Gao, Paul Liang, Yiannis Aloimonos
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MotorSense, a large-scale paired egocentric video and surface EMG dataset (25 hours of unscripted activity) for learning motoric representations of hand action.
Problem
Computer vision models learn hand action only from its visual consequences, lacking access to the underlying embodied motor signals (muscle activity) that produce the actions.
Approach
A paired dataset pairing bimanual surface EMG bands with Meta Aria egocentric glasses, providing synchronized video, bimanual EMG, IMU, and audio, to add motor-signal supervision for action understanding.
Key results
- MotorSense provides 25 hours of unscripted paired egocentric video and bimanual EMG, IMU, and audio data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add video-EMG sensorimotor alignment pre-training to the backbone | Hiera-B (K400-MAE init) · Action recognition, frozen backbone (6 benchmarks metrics) | top-1 accuracy (%): EPIC verb 43.99 -> 59.85; MotorSense 20.31 -> 35.02 | Improved |
| add video-EMG alignment pre-training before end-to-end fine-tuning | Hiera-B (K400-MAE init) · Action recognition, end-to-end | top-1 accuracy (%): EPIC noun 52.16 -> 52.25 (+0.09); MotorSense 42.83 -> 47.51 | Improved |
| pre-train on same recordings with RGB-only SimCLR, MoCo or BYOL instead of EMG alignment | Hiera-B (K400-MAE init) · SSv2 action (end-to-end) | top-1 accuracy (%): 67.57 -> 65.82 (SimCLR) | Hurt |
| initialize ArcticNet-SF backbone with MotorSense sensorimotor pre-training instead of ImageNet | ArcticNet-SF ResNet-50 · H2O 3D hand reconstruction | MPJPE / A-MPJPE (mm): 24.28 -> 19.65 MPJPE; 51.20 -> 44.04 A-MPJPE | Improved |
| use sensorimotor pre-training for contact prediction instead of image-level self-supervision (PixPro, DenseCL, PeCLR) | ResNet-50 · MotorSense hand contact | IoU / AUROC (%): IoU 50.59 -> 55.20; AUROC 93.36 -> 97.19 | Improved |
| image-level self-supervision (DenseCL, PeCLR) on MotorSense frames instead of ImageNet init | ResNet-50 · MotorSense hand contact | IoU (%): 48.41 -> 48.56 (DenseCL), 48.30 (PeCLR) | No clear effect |
| condition video diffusion on motor codes via HyperNetwork LoRA instead of video history | CogVideoX-5B-I2V · MotorSense held-out I2V | FVD: 373 -> 304 | Improved |
| add language conditioning derived from EMG/IMU to zero-shot I2V | CogVideoX-5B-I2V zero-shot · MotorSense held-out I2V | FVD: 926 -> 935 | No clear effect |
| use lightweight LoRA conditioning instead of full supervised fine-tuning | CogVideoX-5B-I2V · MotorSense held-out I2V | FVD: 772 -> 304 (50M vs 5.57B trainable params) | Improved |
| align with EMG only instead of EMG+IMU (ablation) | sensorimotor alignment pre-training · EPIC-Kitchens-100 action (frozen) | top-1 accuracy (%): 21.29 -> 16.33 | Hurt |
| align with IMU only instead of EMG+IMU (ablation) | sensorimotor alignment pre-training · MotorSense action (frozen) | top-1 accuracy (%): 35.02 -> 25.48 | Hurt |
| use fewer EMG electrodes per arm (4 or 2 instead of 8) (ablation) | sensorimotor alignment pre-training · MotorSense action (frozen) | top-1 accuracy (%): 35.02 -> 31.57 (4 per arm), 31.82 (2 per arm) | Hurt |
Closest papers by meaning
- From Pose to Muscle: Multimodal Learning for Piano Hand Muscle Electromyography (NeurIPS 2025 · similarity 0.90)
- emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation (NeurIPS 2024 · similarity 0.90)
- Reading Your Actions: Learning Generalizable Action Representations via Pre-training AEMG (CVPR 2026 · similarity 0.90)
- HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing (NeurIPS 2026 · similarity 0.88)
- Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learning (ICLR 2025 · similarity 0.87)
- emg2qwerty: A Large Dataset with Baselines for Touch Typing using Surface Electromyography (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
