Research map / 3D vision / Human pose estimation
Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes egocentric whole-body (body + hand) motion capture from a single head-mounted fisheye camera using FisheyeViT, a pixel-aligned 3D heatmap pose regressor, and uncertainty-guided diffusion motion refinement, trained on a new 840k-frame synthetic dataset EgoWholeBody.
Problem
Estimating full-body and hand motion from a single head-mounted fisheye camera is hard due to fisheye distortion, self-occlusion of body parts, and the lack of datasets with body and hand ground truth.
Approach
FisheyeViT builds ViT tokens from gnomonically undistorted patches sampled per fixed FOV region, a pixel-aligned 3D heatmap regresses joints in uvd space, and a whole-body diffusion prior refines poses by conditioning on low-uncertainty joints via uncertainty-weighted guidance.
Key results
- On the SceneEgo test set, Ours-Refined reaches 57.59 mm MPJPE for body pose versus 89.06 mm for retrained SceneEgo*.
- On the EgoWholeBody test set, Ours-Refined reaches 60.32 mm MPJPE versus 84.21 mm for retrained EgoPW*.
- Hand pose on SceneEgo: Ours-Refined 19.37 mm MPJPE versus 49.66 mm for Hand4Whole.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FisheyeViT and the pixel-aligned 3D heatmap pose regressor | Egocentric body pose estimation · SceneEgo test dataset | MPJPE: 89.06 -> 64.19 | Helped |
| Add uncertainty-aware whole-body diffusion refinement | Ours-Single body pose estimation · SceneEgo test dataset | MPJPE: 64.19 -> 57.59 | Helped |
| Use the proposed egocentric hand pose estimator | Hand pose estimation · SceneEgo test dataset | MPJPE: 49.66 -> 23.63 | Helped |
| Add whole-body diffusion refinement | Ours-Single hand pose estimation · SceneEgo test dataset | MPJPE: 23.63 -> 19.37 | Helped |
| Add EgoWholeBody training data (ablation) | Egocentric body pose estimation · SceneEgo test dataset | MPJPE: 75.10 -> 64.19 | Helped |
| Add FisheyeViT instead of ViT (ablation) | Egocentric body pose estimation · SceneEgo test dataset | MPJPE: 67.36 -> 64.19 | Helped |
| Use the pixel-aligned 3D heatmap regressor instead of the Mo2Cap2 head (ablation) | Egocentric body pose estimation · SceneEgo test dataset | MPJPE: 87.47 -> 64.19 | Helped |
| Use the pixel-aligned 3D heatmap regressor instead of the xR-egopose head (ablation) | Egocentric body pose estimation · SceneEgo test dataset | MPJPE: 116.5 -> 64.19 | Helped |
| Use the pixel-aligned 3D heatmap regressor instead of the SceneEgo head (ablation) | Egocentric body pose estimation · SceneEgo test dataset | MPJPE: 77.73 -> 64.19 | Helped |
| Use diffusion refinement instead of the GlobalEgoMocap optimizer (ablation) | Temporal body pose refinement · SceneEgo test dataset | MPJPE: 69.83 -> 57.59 | Helped |
| Add uncertainty-aware guidance to diffusion refinement (ablation) | Body pose refinement · SceneEgo test dataset | MPJPE: 62.16 -> 57.59 | Helped |
| Use whole-body diffusion instead of separate body-only diffusion (ablation) | Body pose refinement · SceneEgo test dataset | MPJPE: 58.95 -> 57.59 | Helped |
Limitations
- Relies on synthetic training data; real-world hand ground truth for evaluation required a custom multi-view capture of the SceneEgo sequences.
- Refinement uses stochastic diffusion sampling, so results vary across runs (small std reported).
Experimental setup
Datasets: EgoWholeBody EgoPW EgoBody Mixamo TCDHandMocap GRAB SceneEgo GlobalEgoMocap Mo2Cap2
Benchmarks: SceneEgo test dataset EgoWholeBody test dataset GlobalEgoMocap test dataset Mo2Cap2 test dataset
Metrics: MPJPE PA-MPJPE BA-MPJPE
Baselines: EgoPW SceneEgo Hand4Whole Mo2Cap2 xR-egopose GlobalEgoMocap EgoHMR
Builds on: DDPM EDGE Hand4Whole classifier-guided diffusion sampling SceneEgo
Closest papers by meaning
- Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data (NeurIPS 2024 · similarity 0.88)
- REWIND: Real-Time Egocentric Whole-Body Motion Diffusion with Exemplar-Based Identity Conditioning (CVPR 2025 · similarity 0.87)
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.87)
- Estimating Body and Hand Motion in an Ego-sensed World (CVPR 2025 · similarity 0.87)
- Forecasting of 3D Whole-body Human Poses with Grasping Objects (CVPR 2024 · similarity 0.86)
- DiffGrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.