Research map / 3D vision / Human pose estimation
HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HMD-Poser, a lightweight LSTM+Transformer network that estimates full-body pose and position from scalable HMD plus 0-3 IMU inputs, running in real time on a standalone VR headset.
Problem
Real-time, accurate full-body human motion tracking on standalone VR HMDs with sparse observations, where lower-body motion is under-constrained and computing resources are limited.
Approach
A temporal-spatial feature learning network that pairs per-component LSTMs (temporal, carrying hidden state) with Transformer encoders over only the 8 input components (spatial), so the Transformer sequence length is small. It also adds head-relative hand inputs and an online shape head so joint positions use estimated body shape, and supports scalable HMD/IMU input configurations via zero-padding
Key results
- On AMASS protocol1, HMD+3IMUs reaches MPJPE 1.89 cm and MPJRE 1.73 deg, versus 4.54 cm for PIP and 5.03 cm for AvatarJLM.
- HMD-only scenario achieves MPJPE 3.19 cm on protocol1, beating AvatarPoser (5.84), AGRoL (5.73) and AvatarJLM (5.03).
- Runs at 205.7 Hz on an RTX 3080 GPU and 90.0 Hz on PICO 4 HMD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add lower-leg IMUs to HMD-Poser | HMD-Poser · AMASS protocol1 | MPJPE: 3.19 -> 2.27 | Helped |
| add a pelvis IMU to the HMD+2IMUs configuration | HMD-Poser · AMASS protocol1 | MPJPE: 2.27 -> 1.89 | Helped |
| add lower-leg IMUs to HMD-Poser | HMD-Poser · AMASS protocol2 | MPJPE: 5.44 -> 3.68 | Helped |
| add a pelvis IMU to the HMD+2IMUs configuration | HMD-Poser · AMASS protocol2 | MPJPE: 3.68 -> 3.13 | Helped |
| add hand representations relative to the head coordinate frame (ablation) | HMD-Poser · AMASS protocol1, HMD | H-PE: 2.36 -> 1.65 | Helped |
| add hand representations relative to the head coordinate frame (ablation) | HMD-Poser · AMASS protocol1, HMD | MPJPE: 3.43 -> 3.19 | Helped |
| add the shape regression head (ablation) | HMD-Poser · AMASS protocol1, HMD | MPJPE: 5.08 -> 3.19 | Helped |
| add the shape regression head (ablation) | HMD-Poser · AMASS protocol1, HMD | H-PE: 4.25 -> 1.65 | Helped |
| run the model online on PICO 4 instead of offline | HMD-Poser · Real-captured HMD+2IMUs | MPJPE: 6.53 -> 6.55 | No clear effect |
| use synthetic input instead of real sensor data | HMD-Poser · HMD+2IMUs | MPJPE: 6.53 -> 4.75 | Helped |
Surprising findings
- Real-captured HMD+2IMUs performance (MPJPE 6.55 cm online) is notably worse than synthetic-input performance (4.75 cm), with H-PE gap driven by hand-controller calibration drift.
Limitations
- Data-driven approach highly dependent on large-scale training data.
- IMU limitations make it hard to disambiguate lower-body poses with similar measurements, such as slowly lifting one foot vertically.
- Real-world hand-controller calibration errors cause higher H-PE than synthetic results.
Experimental setup
Datasets: AMASS CMU BMLr HDM05 HumanEva Transition
Benchmarks: AMASS protocol1 AMASS protocol2 free-dancing real-captured dataset
Metrics: MPJRE MPJPE MPJVE Jitter H-PE U-PE L-PE R-PE FPS
Baselines: AvatarPoser AGRoL AvatarJLM TransPose PIP
Builds on: HMD-NeMo AvatarPoser AGRoL PIP TransPose
Closest papers by meaning
- EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUs (AAAI 2025 · similarity 0.87)
- FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video (CVPR 2025 · similarity 0.87)
- Real-Time Simulated Avatar from Head-Mounted Sensors (CVPR 2024 · similarity 0.86)
- Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data (NeurIPS 2024 · similarity 0.85)
- Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input (CVPR 2025 · similarity 0.85)
- EnvPoser: Environment-aware Realistic Human Motion Estimation from Sparse Observations with Uncertainty Modeling (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.