AI Paper Atlas

Research map / Video / Action recognition

MotorSense: A Video-EMG Dataset of Motoric Representations for Action Understanding

Eadom T Dessalene, Michael Maynord, Amir Hossein Shahidzadeh, Botao He, Yasamin Tabatabaee, Ruoshi Liu, Ruohan Gao, Paul Liang, Yiannis Aloimonos

NeurIPS 2026 · Video › Action recognition · approach: Disentangled action recognition · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MotorSense, a large-scale paired egocentric video and surface EMG dataset (25 hours of unscripted activity) for learning motoric representations of hand action.

Problem

Computer vision models learn hand action only from its visual consequences, lacking access to the underlying embodied motor signals (muscle activity) that produce the actions.

Approach

A paired dataset pairing bimanual surface EMG bands with Meta Aria egocentric glasses, providing synchronized video, bimanual EMG, IMU, and audio, to add motor-signal supervision for action understanding.

Key results

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add video-EMG sensorimotor alignment pre-training to the backboneHiera-B (K400-MAE init) · Action recognition, frozen backbone (6 benchmarks metrics)top-1 accuracy (%): EPIC verb 43.99 -> 59.85; MotorSense 20.31 -> 35.02Improved
add video-EMG alignment pre-training before end-to-end fine-tuningHiera-B (K400-MAE init) · Action recognition, end-to-endtop-1 accuracy (%): EPIC noun 52.16 -> 52.25 (+0.09); MotorSense 42.83 -> 47.51Improved
pre-train on same recordings with RGB-only SimCLR, MoCo or BYOL instead of EMG alignmentHiera-B (K400-MAE init) · SSv2 action (end-to-end)top-1 accuracy (%): 67.57 -> 65.82 (SimCLR)Hurt
initialize ArcticNet-SF backbone with MotorSense sensorimotor pre-training instead of ImageNetArcticNet-SF ResNet-50 · H2O 3D hand reconstructionMPJPE / A-MPJPE (mm): 24.28 -> 19.65 MPJPE; 51.20 -> 44.04 A-MPJPEImproved
use sensorimotor pre-training for contact prediction instead of image-level self-supervision (PixPro, DenseCL, PeCLR)ResNet-50 · MotorSense hand contactIoU / AUROC (%): IoU 50.59 -> 55.20; AUROC 93.36 -> 97.19Improved
image-level self-supervision (DenseCL, PeCLR) on MotorSense frames instead of ImageNet initResNet-50 · MotorSense hand contactIoU (%): 48.41 -> 48.56 (DenseCL), 48.30 (PeCLR)No clear effect
condition video diffusion on motor codes via HyperNetwork LoRA instead of video historyCogVideoX-5B-I2V · MotorSense held-out I2VFVD: 373 -> 304Improved
add language conditioning derived from EMG/IMU to zero-shot I2VCogVideoX-5B-I2V zero-shot · MotorSense held-out I2VFVD: 926 -> 935No clear effect
use lightweight LoRA conditioning instead of full supervised fine-tuningCogVideoX-5B-I2V · MotorSense held-out I2VFVD: 772 -> 304 (50M vs 5.57B trainable params)Improved
align with EMG only instead of EMG+IMU (ablation)sensorimotor alignment pre-training · EPIC-Kitchens-100 action (frozen)top-1 accuracy (%): 21.29 -> 16.33Hurt
align with IMU only instead of EMG+IMU (ablation)sensorimotor alignment pre-training · MotorSense action (frozen)top-1 accuracy (%): 35.02 -> 25.48Hurt
use fewer EMG electrodes per arm (4 or 2 instead of 8) (ablation)sensorimotor alignment pre-training · MotorSense action (frozen)top-1 accuracy (%): 35.02 -> 31.57 (4 per arm), 31.82 (2 per arm)Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy