Research map / 3D vision / Human pose estimation
HumanMM: Global Human Motion Recovery from Multi-shot Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HumanMM, a framework that recovers world-coordinate SMPL human motion from multi-shot in-the-wild videos by detecting shot transitions, aligning orientation via epipolar camera calibration, and refining with a multi-shot transformer and trajectory refiner.
Problem
Recovering long-sequence 3D human motion in world coordinates from in-the-wild videos containing abrupt shot transitions, partial occlusions, and dynamic backgrounds, where existing HMR methods assume single-shot continuity.
Approach
Combine a multi-cue shot transition detector (SceneDetect, bbox IoU, keypoint IoU), a masked LEAP-VO that excludes human pixels from bundle adjustment, an orientation alignment module using essential-matrix camera rotation from human 2D keypoints, and a transformer (ms-HMR) with shot-index encoding to align poses across shots, followed by an LSTM trajectory predictor and foot-sliding refiner.
Key results
- On ms-Motion, HumanMM achieves the best PA-MPJPE, WA-MPJPE, RTE and ROE across 2-, 3- and 4-shot videos on ms-AIST and ms-H3.6M (e.g., 2-shot ms-AIST PA-MPJPE 36.82 vs 60.72 for GVHMR).
- The full shot transition detector (SceneDetect + bbox + pose tracking) reaches 0.96 recall, 0.88 precision, 0.92 F1 on ms-Motion, versus 0.70 F1 for SceneDetect alone.
- Masked LEAP-VO reduces RPE translation on EMDB-1 to 0.92 from 1.85 for DPVO (about 50% lower).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HumanMM instead of GVHMR for multi-shot recovery | ms-AIST and ms-H3.6M · 2-Shot and 4-Shot | PA-MPJPE: 60.72 -> 36.82; 64.63 -> 40.52; 102.88 -> 39.63; 90.50 -> 50.59 | Helped |
| Use HumanMM instead of strongest baseline | ms-AIST and ms-H3.6M · 2-Shot and 4-Shot | WA-MPJPE: 231.36 -> 121.35; 254.30 -> 132.13; 603.93 -> 161.52; 512.66 -> 147.62 | Helped |
| Use HumanMM instead of strongest baseline | ms-AIST and ms-H3.6M · 2-Shot and 4-Shot | RTE: 4.39 -> 2.56; 11.41 -> 3.65; 5.57 -> 4.55; 12.91 -> 6.20 | Helped |
| Use HumanMM instead of strongest baseline | ms-AIST and ms-H3.6M · 2-Shot and 4-Shot | ROE: 84.48 -> 69.23; 82.42 -> 53.39; 90.07 -> 70.31; 90.34 -> 61.22 | Helped |
| Add ms-HMR to the motion pipeline (ablation) | HumanMM · EMDB-1 | PA-MPJPE: 78.24 -> 50.49 | Helped |
| Add the orientation alignment module (ablation) | HumanMM · EMDB-1 | ROE: 76.74 -> 47.68 | Helped |
| Add the trajectory predictor (ablation) | HumanMM · EMDB-1 | RTE: 5.52 -> 3.54 | Helped |
| Add the trajectory refiner (ablation) | HumanMM · EMDB-1 | Foot Sliding: 7.84 -> 3.28 | Helped |
| Add bbox tracking to scene transition detection | SceneDetect · ms-Motion | F1 Score: 0.70 -> 0.86 | Helped |
| Add pose tracking to scene and bbox transition detection | SceneDetect + bbox tracking · ms-Motion | F1 Score: 0.86 -> 0.92 | Helped |
Limitations
- Performance may decline when there are an excessive number of shot transitions.
- Evaluation relies on a self-constructed multi-shot dataset synthesized by splicing single-shot mocap clips.
Experimental setup
Datasets: AMASS 3DPW Human3.6M BEDLAM AIST EMDB
Benchmarks: ms-Motion EMDB-1 EMDB-2
Metrics: PA-MPJPE WA-MPJPE RTE ROE Jitter Foot sliding Recall Precision F1 Score ATE
Baselines: SLAHMR WHAM GVHMR DPVO LEAP-VO
Builds on: GVHMR LEAP-VO WHAM DPVO SAM
Closest papers by meaning
- Synergistic Global-space Camera and Human Reconstruction from Videos (CVPR 2024 · similarity 0.89)
- WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion (CVPR 2024 · similarity 0.89)
- Reconstructing People, Places, and Cameras (CVPR 2025 · similarity 0.88)
- ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos (CVPR 2025 · similarity 0.88)
- Motions as Queries: One-Stage Multi-Person Holistic Human Motion Capture (CVPR 2025 · similarity 0.88)
- HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.