Research map / 3D vision / Human pose estimation
Reconstructing People, Places, and Cameras
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Jointly reconstructs multiple SMPL-X human meshes, scene pointmaps, and cameras in a metric world frame from sparse uncalibrated multi-view images, by optimizing bundle adjustment on 2D human keypoints plus a DUSt3R-style global alignment loss.
Problem
Reconstructing people, scene structure, and cameras jointly in a common metric world coordinate system from sparse uncalibrated multi-view images; existing scene/SfM methods lack metric scale and human pose methods lack scene anchoring.
Approach
Integrates human mesh predictions and 2D human keypoint correspondences into SfM: human body size provides an approximate metric scale for the data-driven scene (DUSt3R) reconstruction, and joint optimization of humans, pointmaps, and cameras improves all three components.
Key results
- On EgoHumans, world-frame human error (W-MPJPE) drops from 3.51m (UnCaliPose) to 1.04m with HSfM.
- On EgoExo4D, W-MPJPE drops from 3.59m to 0.50m.
- On EgoHumans, joint human-scene-camera optimization raises RRA@15 by about 20% relative to initialization, and HSfM reduces camera translation error from 4.97m (MASt3R) to 2.09m.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly optimize human meshes, scene, and cameras | HSfM · EgoHumans | W-MPJPE (m): 4.28 -> 1.04 | Helped |
| jointly optimize human meshes, scene, and cameras | HSfM · EgoHumans | RRA@15: 0.79 -> 0.89 | Helped |
| jointly optimize human meshes, scene, and cameras | HSfM · EgoExo4D | W-MPJPE (m): 5.80 -> 0.50 | Helped |
| detach human-loss gradients from scene and camera parameters (ablation) | HSfM · EgoHumans | W-MPJPE (m): 1.04 -> 3.94 | Hurt |
| remove the global scene loss (ablation) | HSfM · EgoHumans | W-MPJPE (m): 1.04 -> 1.29 | Hurt |
| remove the global scene loss (ablation) | HSfM · EgoHumans | RRA@15: 0.89 -> 0.73 | Hurt |
| use 1 human rather than 4 in optimization (ablation) | HSfM · EgoHumans | W-MPJPE (m): 1.28 -> 1.69 | Hurt |
| increase optimized people from 1 to 4 (ablation) | HSfM · EgoHumans | TE (m): 1.91 -> 1.52 | Helped |
| use HSfM instead of UnCaliPose | human world localization · EgoHumans | W-MPJPE (m): 3.51 -> 1.04 | Helped |
| use HSfM instead of UnCaliPose | human world localization · EgoExo4D | W-MPJPE (m): 3.59 -> 0.50 | Helped |
Surprising findings
- Optimizing cameras and humans with only the human keypoint loss (no scene loss) improves human world localization but degrades camera metrics, e.g. CCA@15 down 36.8%.
Limitations
- Optimization-based framework that can be sensitive to hyper-parameters.
- Scene reconstruction still struggles on uneven ground planes.
- Assumes known person re-identification across views.
Experimental setup
Datasets: EgoHumans EgoExo4D
Benchmarks: EgoHumans EgoExo4D
Metrics: W-MPJPE GA-MPJPE PA-MPJPE TE s-TE AE RRA CCA s-CCA
Baselines: UnCaliPose DUSt3R MASt3R HSfM (init.) Multi-HMR SLAHMR
Builds on: DUSt3R SMPL-X HMR2 ViTPose UnCaliPose
Closest papers by meaning
- ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos (CVPR 2025 · similarity 0.91)
- Synergistic Global-space Camera and Human Reconstruction from Videos (CVPR 2024 · similarity 0.90)
- Guess The Unseen: Dynamic 3D Scene Reconstruction from Partial 2D Glimpses (CVPR 2024 · similarity 0.89)
- HumanMM: Global Human Motion Recovery from Multi-shot Videos (CVPR 2025 · similarity 0.88)
- MultiPly: Reconstruction of Multiple People from Monocular Video in the Wild (CVPR 2024 · similarity 0.88)
- UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.