Research map / 3D vision / Human pose estimation
Multiple View Geometry Transformers for 3D Human Pose Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MVGFormer, a hybrid transformer that iteratively refines compositional pose queries with a learned 2D appearance module and a learning-free triangulation geometry module, for multi-view 3D human pose estimation with better cross-camera generalization.
Problem
Multi-view 3D human pose estimation: end-to-end learned transformers and voxel methods overfit to training camera configurations and fail on unseen cameras and datasets, while geometric triangulation methods struggle under occlusion.
Approach
Decoder layers alternate a learned appearance module (projective deformable attention refining 2D poses per view) with a learning-free geometry module (differentiable algebraic triangulation of refined 2D poses), so queries are iteratively refined in a coarse-to-fine manner with explicit multi-view geometry.
Key results
- In-domain on CMU Panoptic, MVGFormer reaches AP25 92.3 and MPJPE 16.0 mm, matching MvP (92.3) and beating VoxelPose (84.0 AP25).
- Cross-dataset without finetuning (CMU0 to Shelf), MVGFormer reaches 87.9% PCP versus 69.8% for VoxelPose and 8.7% for MvP.
- Cross-camera-arrangement on CMU1-4, MVGFormer achieves 74.7 average AP25 versus 29.4 for VoxelPose and 0.0 for MvP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase initialized query count from 128 to 1024 (ablation) | MVGFormer · CMU Panoptic ablation | AP25: 77.9 -> 92.3 | Helped |
| Increase initialized query count from 1024 to 1280 (ablation) | MVGFormer · CMU Panoptic ablation | AP25: 92.3 -> 92.0 | Hurt |
| Use four decoder layers instead of one (ablation) | MVGFormer · CMU Panoptic ablation | AP25: 5.2 -> 92.3 | Helped |
| Increase decoder layers from four to five (ablation) | MVGFormer · CMU Panoptic ablation | AP25: 92.3 -> 91.0 | Hurt |
| Increase query assignment from 1 to 5 nearest queries per ground-truth pose (ablation) | MVGFormer · CMU Panoptic ablation | AP25: 86.0 -> 92.3 | Helped |
| Use person-level appearance embeddings instead of joint-level embeddings (ablation) | MVGFormer · CMU Panoptic ablation | AP25: 90.9 -> 92.3 | Helped |
| Replace MLP feature fusion with attention (ablation) | MVGFormer · CMU Panoptic ablation | AP25: 92.3 -> 90.4 | Hurt |
| Replace MLP feature fusion with mean fusion (ablation) | MVGFormer · CMU Panoptic ablation | AP25: 92.3 -> 36.2 | Hurt |
| Add non-maximum suppression to remove redundant pose estimates (ablation) | MVGFormer · CMU Panoptic ablation | AP25: 20.0 -> 92.3 | Helped |
| Use MVGFormer rather than VoxelPose for cross-camera evaluation | CMU0 camera-count changes · CMU0 (3), cross-camera | AP25: 13.7 -> 44.6 | Helped |
| Use MVGFormer rather than VoxelPose for cross-arrangement evaluation | CMU0-trained models · CMU1, cross-arrangement | AP25: 28.8 -> 86.8 | Helped |
| Use MVGFormer rather than VoxelPose without finetuning | CMU0-trained models · Shelf, cross-dataset | PCP: 69.8 -> 87.9 | Helped |
Surprising findings
- MvP's accuracy drops to essentially zero on unseen camera arrangements and datasets, while the learned-prior approaches overfit to training cameras.
- The Hungarian one-to-one matching loss from DETR failed to converge in this setting, so a KNN multiple-to-one assignment was used instead.
Limitations
- Performance on Campus without finetuning is relatively low (58.1 PCP) due to appearance domain gap and larger outdoor space.
- Accuracy on CMU2 and CMU3 arrangements is lower due to few cameras and missing views.
- Single-frame only; tracking left as future work.
Experimental setup
Datasets: CMU Panoptic Shelf Campus COCO
Benchmarks: CMU Panoptic Shelf Campus
Metrics: AP25 AP100 mAP MPJPE PCP Recall
Baselines: VoxelPose MvP Dong et al. Faster VoxelPose VoxelTrack Graph3D
Builds on: DETR deformable attention VoxelPose MvP differentiable algebraic triangulation
Closest papers by meaning
- MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation (CVPR 2022 · similarity 0.89)
- SVTformer: Spatial-View-Temporal Transformer for Multi-View 3D Human Pose Estimation (AAAI 2025 · similarity 0.88)
- MV-SSM: Multi-View State Space Modeling for 3D Human Pose Estimation (CVPR 2025 · similarity 0.87)
- FusionFormer: A Concise Unified Feature Fusion Transformer for 3D Pose Estimation (AAAI 2024 · similarity 0.86)
- PoseIRM: Enhance 3D Human Pose Estimation on Unseen Camera Settings via Invariant Risk Minimization (CVPR 2024 · similarity 0.86)
- MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.