Research map / 3D vision / Depth and stereo
Synergistic Global-space Camera and Human Reconstruction from Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SynCHMR, which calibrates monocular depth with camera-frame human meshes to get metric-scale SLAM, then denoises world-frame SMPL trajectories conditioned on the reconstructed dense scene, yielding consistent camera, human, and scene reconstructions in one global frame.
Problem
Jointly reconstructing metric-scale camera trajectories, human meshes, and dense scene geometry in a common world frame from monocular videos with moving cameras, where SLAM is scale-ambiguous and dynamic humans break static assumptions.
Approach
Use camera-frame HMR as a metric prior to calibrate monocular depth and mask dynamic humans inside DROID-SLAM (Human-aware Metric SLAM), then train a scene-conditioned transformer denoiser that refines world-frame SMPL tracks using the dense dynamic point cloud as implicit scene constraint.
Key results
- On EgoBody validation, SynCHMR reduces WA-MPJPE to 81.1 mm versus 101.2 mm for SLAHMR with PHALP+ and FA-MPJPE to 115.1 mm versus 141.1 mm.
- On EgoBody test, SynCHMR achieves WA-MPJPE 84.6 mm and FA-MPJPE 122.1 mm, improving over SLAHMR w/ PHALP+ (99.4 and 163.9 mm).
- Calibrated depth with foreground masking cuts EMDB camera ATE from 385.8 to 107.0 relative to uncalibrated ZoeDepth+ with Mask2Former baseline (389.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add human-aware metric SLAM to DROID-SLAM camera estimates | SLAHMR with 4DHumans · 3DPW-Test | PA-MPJPE: 57.4 -> 52.9 | Helped |
| add the Scene-aware SMPL Denoiser (ablation) | Human-aware Metric SLAM + 4DHumans · 3DPW-Test | PA-MPJPE: 52.9 -> 52.4 | Helped |
| add the Scene-aware SMPL Denoiser (ablation) | Human-aware Metric SLAM + 4DHumans · EgoBody Val | PA-MPJPE: 73.0 -> 57.7 | Helped |
| add calibrated ZoeDepth+ depths and foreground masks (ablation) | Human-aware Metric SLAM · EgoBody | ATE: 80.9 -> 26.4 | Helped |
| add calibrated ZoeDepth+ depths and foreground masks (ablation) | Human-aware Metric SLAM · EMDB | ATE: 400.3 -> 107.0 | Helped |
| add ZoeDepth+ without depth calibration (ablation) | RGB-only SLAM · EgoBody | ATE: 80.9 -> 35.0 | Helped |
| add Mask2Former masks without depth (ablation) | RGB-only SLAM · EgoBody | ATE: 80.9 -> 81.6 | Hurt |
| add XYZ features to ViT + RGB conditioning (ablation) | Scene-aware SMPL Denoiser · EgoBody Val | PA: 63.9 -> 64.5 | Hurt |
| add subject masks to ViT + RGB conditioning (ablation) | Scene-aware SMPL Denoiser · EgoBody Val | PA: 63.9 -> 66.8 | Hurt |
| add RGB features to SPVCNN + XYZ conditioning (ablation) | Scene-aware SMPL Denoiser · EgoBody Val | PA: 62.9 -> 61.0 | Helped |
| add subject masks to SPVCNN + XYZ conditioning (ablation) | Scene-aware SMPL Denoiser · EgoBody Val | PA: 62.9 -> 62.0 | Hurt |
| use SPVCNN instead of ViT for scene encoding with RGB, XYZ, and masks (ablation) | Scene-aware SMPL Denoiser · EgoBody Val | PA: 69.3 -> 61.3 | Helped |
Surprising findings
- Initializing SLAHMR's global optimization with 4DHumans instead of PHALP+ did not improve results, despite 4DHumans being more accurate per frame.
Limitations
- Calibrating depth with estimated bodies is less reliable for subjects that the body model cannot explain well, such as children or obese people.
- Focal length is approximated as (W+H)/2, not estimated.
- Accuracy on composed or generated videos is untested.
Experimental setup
Datasets: 3DPW EgoBody EMDB DAVIS
Benchmarks: 3DPW EgoBody EMDB DAVIS
Metrics: PA-MPJPE FA-MPJPE WA-MPJPE acceleration error ATE threshold accuracy REL RMSE
Baselines: GLAMR SLAHMR PACE TRACE DROID-SLAM
Builds on: DROID-SLAM 4DHumans ZoeDepth SLAHMR SMPL
Closest papers by meaning
- EIHMR: Collaborative Human-Camera Estimation for Global Human Mesh Recovery (NeurIPS 2026 · similarity 0.91)
- Reconstructing People, Places, and Cameras (CVPR 2025 · similarity 0.90)
- HumanMM: Global Human Motion Recovery from Multi-shot Videos (CVPR 2025 · similarity 0.89)
- ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos (CVPR 2025 · similarity 0.89)
- UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.88)
- SLAM3R: Real-Time Dense Scene Reconstruction from Monocular RGB Videos (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.