Research map / 3D vision / Human pose estimation
Estimating Body and Hand Motion in an Ego-sensed World
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents EgoAllo, which samples full-body SMPL-H pose, height, and hand motion from a conditional diffusion prior guided by egocentric SLAM head poses, with a locally invariant head-pose conditioning parameterization.
Problem
Estimating a head-mounted device wearer's 3D body pose, height, and hand motion in the allocentric world frame from egocentric SLAM poses and video, where most body parameters are unobserved.
Approach
Derive spatial and temporal invariance criteria for head-pose conditioning and propose a per-timestep locally canonicalized relative head-motion representation that satisfies both; combine the conditional diffusion prior with Levenberg-Marquardt guidance from HaMeR hand detections, skating, and prior costs.
Key results
- Invariant conditioning (Eq. 4) reduces MPJPE by 17.9% vs sequence canonicalization and 23.2% vs absolute conditioning on AMASS length-32 sequences.
- EgoAllo reduces MPJPE vs EgoEgo by roughly 20-30% across datasets (e.g., 129.8mm vs 184.0mm on AMASS length 32).
- EgoAllo-Mono reduces hand world-frame MPJPE from 237.90mm (HaMeR) to 131.45mm, and EgoAllo-Wrist3D to 60.08mm on EgoExo4D.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use invariant conditioning instead of sequence canonicalization (ablation) | EgoAllo motion prior · AMASS test, sequence length 32 | MPJPE: 153.1±1.5 -> 129.8±1.1 | Helped |
| use invariant conditioning instead of sequence canonicalization (ablation) | EgoAllo motion prior · AMASS test, sequence length 128 | MPJPE: 134.0±1.8 -> 119.7±1.3 | Helped |
| use invariant conditioning instead of absolute-pose conditioning (ablation) | EgoAllo motion prior · AMASS test, sequence length 32 | MPJPE: 159.9±1.2 -> 129.8±1.1 | Helped |
| use invariant conditioning instead of absolute-pose conditioning (ablation) | EgoAllo motion prior · AMASS test, sequence length 128 | MPJPE: 148.3±1.5 -> 119.7±1.3 | Helped |
| add shape estimation (ablation) | EgoAllo body estimation · AMASS, sequence length 32 | MPJPE: 138.1±1.1 -> 129.8±1.1 | Helped |
| add shape estimation (ablation) | EgoAllo body estimation · AMASS, sequence length 128 | MPJPE: 128.1±1.3 -> 119.7±1.3 | Helped |
| add reprojection loss to hand guidance (ablation) | EgoAllo hand estimation · EgoExo4D validation | MPJPE: 143.20±0.42 -> 131.45±0.39 | Helped |
| add reprojection loss to hand guidance (ablation) | EgoAllo hand estimation · EgoExo4D validation | PA-MPJPE: 14.75±0.42 -> 14.71±0.39 | No clear effect |
| add Project Aria 3D wrist pose losses (ablation) | EgoAllo hand estimation · EgoExo4D validation | MPJPE: 131.45mm→60.08mm | Helped |
| add Project Aria 3D wrist pose losses (ablation) | EgoAllo hand estimation · EgoExo4D validation | PA-MPJPE: 14.71±0.39 -> 14.38±0.26 | No clear effect |
| use body-conditioned hand-body estimation instead of single-frame HaMeR | EgoAllo-Mono · EgoExo4D validation | MPJPE: 237.90mm →131.45mm | Helped |
Surprising findings
- Sequence canonicalization from EgoEgo, a common approach, underperforms even simple absolute conditioning on some metrics at length 128 for PA-MPJPE relative to the proposed method, showing temporal invariance matters.
- Shape estimation gives accurate height (52mm to 32mm mean height error) but does not generalize for body weight (5kg to 8kg error).
Limitations
- Diffusion guidance is test-time optimization that depends on hyperparameters and incurs runtime cost.
- Hand guidance depends on reasonable monocular HaMeR estimates; left/right flips and spurious detections cause failures.
- Assumes flat floors; fails on hills or staircases.
Experimental setup
Datasets: AMASS RICH Aria Digital Twins EgoExo4D
Benchmarks: AMASS RICH Aria Digital Twins EgoExo4D
Metrics: MPJPE PA-MPJPE GND Thead hand MPJPE
Baselines: EgoEgo VAE+Opt NoShape HaMeR EgoPoser AvatarPoser
Builds on: EgoEgo HuMoR DDIM MultiDiffusion SLAHMR
Closest papers by meaning
- Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data (NeurIPS 2024 · similarity 0.91)
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.88)
- HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos (CVPR 2025 · similarity 0.88)
- FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video (CVPR 2025 · similarity 0.87)
- HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos (CVPR 2025 · similarity 0.87)
- StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Video (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.