Research map / 3D vision / Human pose estimation
Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DSPoser, which estimates full-body ego pose from head tracking plus temporally sparse hand poses detected in egocentric video, using an uncertainty-aware masked autoencoder for hand imputation and an uncertainty-guided VQ-Diffusion for body generation.
Problem
Estimating full-body pose of an HMD camera wearer when hands are only intermittently visible in egocentric video, so hand signals are temporally sparse on top of spatially sparse head-plus-hand inputs.
Approach
Decompose into temporal completion (uncertainty-aware MAE imputes invisible hand trajectories from head pose and visible hands, with aleatoric/epistemic uncertainty from an ensemble of 4 models trained with beta-NLL) and spatial completion (VQ-Diffusion generates full-body motion conditioned on imputed hands, head, and uncertainty guidance via sampling, dropout, or distribution embedding).
Key results
- On AMASS with T_s=20 and doubly sparse input, DSPoser reaches MPJPE 5.51 cm versus 7.35 cm for MAE-imputed BoDiffusion.
- On the Ego-Exo4D validation set (T_s=20), DSPoser achieves MPJPE 16.84 cm, improving on the best baseline (naive BoDiffusion extension with MAE, 22.12 cm) by 5.28 cm.
- On AMASS with T_s=1, DSPoser reaches MPJPE 5.87 cm, beating AvatarJLM with MAE (7.12 cm) but with higher MPJVE (49.12 vs 37.60).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MAE-imputed sparse hand trajectories to head tracking | DSPoser · AMASS test, Ts=20 | MPJPE: 12.08 -> 5.51 | Helped |
| Use DSPoser rather than BoDiffusion with MAE imputation | BoDiffusion + MAE · AMASS test, Ts=20 | MPJPE: 7.35 -> 5.51 | Helped |
| Add MAE-imputed sparse hand trajectories to head tracking | DSPoser · Ego-Exo4D validation, Ts=20 | MPJPE: 19.12 -> 16.84 | Helped |
| Remove uncertainty guidance (ablation) | DSPoser · AMASS test, Ts=20 | MPJPE: 6.05 -> 5.51 | Hurt |
| Use distribution embedding instead of sampling (ablation) | DSPoser · AMASS test, Ts=20 | MPJPE: 5.51 -> 5.67 | Hurt |
| Use dropout instead of sampling for uncertainty guidance (ablation) | DSPoser · AMASS test, Ts=20 | MPJPE: 5.51 -> 5.55 | Hurt |
| Use epistemic uncertainty instead of aleatoric uncertainty (ablation) | DSPoser · AMASS test, Ts=20 | MPJPE: 5.51 -> 5.78 | Hurt |
| Use total uncertainty instead of aleatoric uncertainty (ablation) | DSPoser · AMASS test, Ts=20 | MPJPE: 5.51 -> 5.59 | Hurt |
| Set MAE uncertainty parameter beta to 0.50 (ablation) | MAE temporal completion · AMASS | MPJPE: 11.57 -> 10.85 | Helped |
Surprising findings
- Naively applying linear interpolation to temporally sparse hand poses degrades BoDiffusion badly (MPJPE 46.45 cm on AMASS), worse than head-only inputs.
- Diffusion-based outputs show higher MPJVE than baselines due to sampling stochasticity, a trade-off the authors acknowledge.
Limitations
- Only hands are used as sparse body-part signals; feet or elbows are not explored.
- Evaluated only on short windows (a few seconds); larger windows untested.
- Diffusion inference is slow and epistemic uncertainty requires multiple models, limiting real-time use.
Experimental setup
Datasets: AMASS Ego-Exo4D
Benchmarks: AMASS Ego-Exo4D
Metrics: MPJPE MPJVE MPJRE
Baselines: EgoEgo BoDiffusion AvatarPoser AvatarJLM FinalIK LoBSTR VAE-HMD CollMoves
Builds on: MAE VQ-Diffusion BoDiffusion beta-NLL T2M-GPT VQ-VAE
Closest papers by meaning
- Estimating Body and Hand Motion in an Ego-sensed World (CVPR 2025 · similarity 0.91)
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.90)
- REWIND: Real-Time Egocentric Whole-Body Motion Diffusion with Exemplar-Based Identity Conditioning (CVPR 2025 · similarity 0.89)
- Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement (CVPR 2024 · similarity 0.88)
- Whole-Body Conditioned Egocentric Video Prediction (NeurIPS 2025 · similarity 0.88)
- FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.