Research map / 3D vision / Human pose estimation
EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EMHI, a real-world multimodal egocentric motion dataset pairing stereo downward-sloping headset cameras with body-worn IMUs and SMPL annotations, plus a baseline MEPoser that fuses image and IMU features for egocentric pose estimation.
Problem
Egocentric human pose estimation for VR/AR suffers from self-occlusion in images and drift/sparsity in IMUs, and there is a lack of real-world datasets containing both modalities captured on real VR hardware.
Approach
A large real-world dataset captured with a commercial VR headset (stereo downward-sloping cameras, 6DoF head/hands, 3DoF legs) with SMPL labels from multi-view markerless capture plus IMU-based rotation constraints, and a lightweight LSTM-based fusion model (MEPoser) that combines stereo image heatmaps, 3D joints and IMU features for real-time estimation on a standalone HMD.
Key results
- On EMHI Protocol 1, MEPoser-Full reaches 3.7 cm MPJPE versus 5.5 cm for UnrealEgo and 5.8 cm for HMD-Poser.
- On EMHI Protocol 2, MEPoser-Full reaches 4.8 cm MPJPE, a 25% reduction relative to UnrealEgo's 6.4 cm.
- Cross-validation of annotations against optical marker-based SMPL fitting gives an average MPJPE of 2.51 cm across 9 sequences.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine egocentric vision and IMU instead of vision alone (ablation) | MEPoser · EMHI Protocol 1 | MPJPE (cm): 4.5 -> 3.7 | Helped |
| combine egocentric vision and IMU instead of vision alone (ablation) | MEPoser · EMHI Protocol 2 | MPJPE (cm): 5.4 -> 4.8 | Helped |
| combine egocentric vision and IMU instead of IMU alone (ablation) | MEPoser · EMHI Protocol 1 | MPJPE (cm): 6.2 -> 3.7 | Helped |
| combine egocentric vision and IMU instead of IMU alone (ablation) | MEPoser · EMHI Protocol 2 | MPJPE (cm): 7.1 -> 4.8 | Helped |
| combine egocentric vision and IMU instead of vision alone (ablation) | MEPoser · EMHI Protocol 1 | Jitter: 511.0 -> 161.8 | Hurt |
| combine egocentric vision and IMU instead of vision alone (ablation) | MEPoser · EMHI Protocol 2 | Jitter: 566.5 -> 204.9 | Hurt |
| combine egocentric vision and IMU instead of IMU alone (ablation) | MEPoser · EMHI Protocol 1 | Jitter: 121.7 -> 161.8 | Hurt |
| combine egocentric vision and IMU instead of IMU alone (ablation) | MEPoser · EMHI Protocol 2 | Jitter: 161.7 -> 204.9 | Hurt |
| combine egocentric vision and IMU | MEPoser · EMHI Protocol 1 | MPJPE (cm): 5.5 -> 3.7 | Helped |
| combine egocentric vision and IMU | MEPoser · EMHI Protocol 2 | MPJPE (cm): 6.4 -> 4.8 | Helped |
| combine egocentric vision and IMU | MEPoser · EMHI Protocol 1 | MPJRE: 4.6 -> 4.1 | Helped |
| combine egocentric vision and IMU | MEPoser · EMHI Protocol 2 | MPJRE: 4.9 -> 4.7 | Helped |
Surprising findings
- The image-only MEPoser-CV variant has much higher jitter (511.0) than the IMU-based HMD-Poser (114.9), so the temporal smoothness benefit comes mainly from the IMU branch.
Limitations
- Annotation pipeline relies on IMU-derived rotations for occluded wrists and head, and the cross-validation covers only 9 sequences.
- Image-only and IMU-only variants show high jitter or lower accuracy, so the fused model's gains depend on both modalities being available.
Experimental setup
Datasets: EMHI
Benchmarks: EMHI Protocol 1 EMHI Protocol 2
Metrics: MPJRE MPJPE PA-MPJPE UpperPE LowerPE RootPE Jitter
Baselines: UnrealEgo HMD-Poser EgoPoseFormer AvatarPoser AvatarJLM
Builds on: HMD-Poser UnrealEgo SMPL HuMMan RegNet
Closest papers by meaning
- FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video (CVPR 2025 · similarity 0.89)
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.89)
- Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input (CVPR 2025 · similarity 0.89)
- HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations (CVPR 2024 · similarity 0.87)
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.86)
- Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.