Research map / 3D vision / Human pose estimation
Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Lightweight full-body motion capture from two smartwatch IMUs plus a head-mounted monocular camera, using SLAM-derived head poses with floor-level tracking, a multi-stage Transformer regressor, and egocentric-visual-cue motion optimization.
Problem
Full-body motion capture typically requires 6 or more expert IMUs or VR tracking devices, which limits accessibility and restricts capture to controlled indoor settings; sparse wrist-only signals are highly ambiguous.
Approach
Integrate 6DoF head poses from a head-mounted monocular SLAM camera into a transformer-based IMU motion regressor, with a floor-level tracking algorithm so head height is correct on uneven terrain, and refine motion on a learned motion manifold using 2D hand keypoints and third-person views from other wearers.
Key results
- On TotalCapture, the method reaches MPJPE 4.27 cm vs 34.69 for PIP and 41.79 for TIP, with root PE 3.25 vs 34.15 and 41.18.
- On HPS, the head-camera system achieves MPJPE 8.65 cm vs 31.47 for AGRoL.
- Floor-level update reduces full-body PE from 12.35 to 6.53 on a stair-walking sequence; visual-cue optimization reduces right-hand PE from 11.91 to 5.55.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add floor-level updating to Fest (ablation) | Fest · Walking downstairs | Full PE: 12.35 -> 6.53 | Helped |
| Add motion optimization using egocentric visual cues (ablation) | Fest · Hand-based everyday interactions | Right Hand PE: 11.91 -> 5.55 | Helped |
| Use two wrist IMUs and head-camera poses in Fest instead of six full-body IMUs | Full-body motion capture · TotalCapture | MPJPE: 34.69 -> 4.27 | Helped |
| Use two wrist IMUs and head-camera poses in Fest instead of six full-body IMUs | Full-body motion capture · TotalCapture | r.MPJPE: 4.40 -> 3.77 | Helped |
| Use Fest rather than AGRoL | Upper-body motion capture · HPS | MPJPE (r.): 31.47 (27.64) -> 8.65 (8.24) | Helped |
| Use Fest rather than AGRoL | Upper-body motion capture · AMASS | MPJVE: 19.31 -> 17.00 | Helped |
Surprising findings
- Using only two smartwatches plus a head camera matches or beats 6-IMU full-body methods on root-relative error, despite fewer sensors.
Limitations
- Relies on off-the-shelf DROID-SLAM, which fails in rare catastrophic cases.
- Networks are trained with a mean body shape; body shape variation is not handled.
- Real-world head poses are adjusted by scaling.
Experimental setup
Datasets: AMASS TotalCapture HPS DIP
Benchmarks: TotalCapture AMASS HPS
Metrics: MPJPE MPJVE Jitter Root PE r.MPJPE
Baselines: PIP TIP AGRoL
Builds on: DROID-SLAM AvatarPoser-style transformer regression TransPose PIP AGRoL
Closest papers by meaning
- Capturing the Unseen: Vision-Free Facial Motion Capture Using Inertial Measurement Units (AAAI 2025 · similarity 0.83)
- Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input (CVPR 2025 · similarity 0.83)
- FreeCap: Hybrid Calibration-Free Motion Capture in Open Environments (AAAI 2025 · similarity 0.83)
- MotionPRO: Exploring the Role of Pressure in Human MoCap and Beyond (CVPR 2025 · similarity 0.83)
- Estimating Body and Hand Motion in an Ego-sensed World (CVPR 2025 · similarity 0.82)
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.