Research map / 3D vision / Human pose estimation
Motions as Queries: One-Stage Multi-Person Holistic Human Motion Capture
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Motions as Queries (MaQ), a one-stage end-to-end network that captures whole-body motions of all people in a monocular video using temporally consistent queries, and introduces the M3C synthetic multi-person dataset with moving cameras.
Problem
Capturing multi-person holistic (whole-body) human motion from monocular video usually requires a cascaded detector, tracker, and HPS estimator, which suffers from error propagation, no end-to-end training, and inference time that scales with the number of people.
Approach
A single DETR-like network where each temporally consistent motion query, cross-attending to per-frame spatiotemporal features, is responsible for one individual's whole motion across the clip, implicitly handling detection, tracking, and SMPL-X estimation in one shot and trainable end-to-end.
Key results
- On BEDLAM, achieves 128 ID switches, 95.15 MOTA, 97.73 HOTA versus 515 IDs and 84.59 HOTA for Multi-HMR + ByteTrack, with inference time 0.027 vs 0.201 s per frame-equivalent unit.
- On Dyna3DPW, reports 0 ID switches and 99.6 MOTA, the best among compared methods on three tracking metrics.
- On 3DPW, PA-MPJPE 44.7 and MPJPE 72.6 for video-based methods without GT boxes, outperforming TRACE (50.8 / 80.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Motions as Queries instead of Multi-HMR + ByteTrack | Multi-person whole-body motion capture · BEDLAM | HOTA: 84.59 -> 97.73 | Helped |
| Use Motions as Queries instead of Multi-HMR + ByteTrack | Multi-person whole-body motion capture · BEDLAM | MPJPE: 81.58 -> 79.88 | Helped |
| Use Motions as Queries instead of Multi-HMR + ByteTrack | Multi-person whole-body motion capture · BEDLAM | PA-MPJPE: 42.93 -> 45.56 | Hurt |
| Use Motions as Queries instead of TRACE | Multi-person motion tracking · Dyna3DPW | HOTA: 74.7 -> 70.8 | Hurt |
| Add 3DPW training data to BEDLAM (ablation) | Motions as Queries · 3DPW test | MPJPE: 82.18 -> 75.86 | Helped |
| Add M3C training data to BEDLAM + 3DPW (ablation) | Motions as Queries · 3DPW test | MPJPE: 75.86 -> 72.62 | Helped |
| Replace dynamic convolution with feature concatenation (ablation) | Motions as Queries · BEDLAM test | MPJPE: 79.88 -> 116.11 | Hurt |
| Estimate body motion in parallel with the bounding-box branch (ablation) | Motions as Queries · BEDLAM test | MPJPE: 79.88 -> 289.28 | Hurt |
Surprising findings
- Multi-HMR + ByteTrack achieves better single-frame pose accuracy than the proposed method on BEDLAM (PA-MPJPE 42.93 vs 45.56), which the authors attribute to broader single-frame training data.
Limitations
- Struggles when individuals are severely occluded for long periods.
- Pose accuracy on single-frame benchmarks trails Multi-HMR, attributed to less training data for the motion task.
- Evaluated mainly on BEDLAM and 3DPW; no existing whole-body motion benchmark.
Experimental setup
Datasets: BEDLAM 3DPW M3C
Benchmarks: BEDLAM 3DPW Dyna3DPW
Metrics: IDs MOTA IDF1 HOTA MPJPE PA-MPJPE PVE inference time
Baselines: Fast-RCNN + SMPLer-X + ByteTrack Multi-HMR + ByteTrack PHALP YOLOX + ByteTrack BEV + ByteTrack TRACE WHAM HybrIK METRO CLIFF OSX ROMP
Builds on: DETR Multi-HMR ByteTrack SMPL-X dynamic convolution
Closest papers by meaning
- HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation (NeurIPS 2024 · similarity 0.89)
- Capturing Closely Interacted Two-Person Motions with Reaction Priors (CVPR 2024 · similarity 0.89)
- CoMotion: Concurrent Multi-person 3D Motion (ICLR 2025 · similarity 0.88)
- HumanMM: Global Human Motion Recovery from Multi-shot Videos (CVPR 2025 · similarity 0.88)
- MultiPhys: Multi-Person Physics-aware 3D Motion Estimation (CVPR 2024 · similarity 0.88)
- I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.