Research map / 3D vision / Human pose estimation
MV-SSM: Multi-View State Space Modeling for 3D Human Pose Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MV-SSM, a multi-view 3D human pose estimator that stacks Projective State Space blocks (projective attention plus Mamba-style SS2D scanning over joint tokens, called GTBS) with progressive triangulation-based refinement, to improve cross-camera and cross-dataset generalization.
Problem
Multi-view 3D human pose estimation generalizes poorly to new camera configurations and scenes, and attention-based transformers overfit training camera arrangements and struggle with occlusion.
Approach
Model the joint spatial sequence with state space models (Mamba/SS2D) over projected, grid-token-guided joint tokens at both feature and person-keypoint levels, stacked in progressive refinement blocks that combine projective attention with bidirectional scanning, followed by differentiable algebraic triangulation.
Key results
- Achieves AP25 93.5 on CMU Panoptic in-domain vs 92.3 for MVGFormer and MvP, with MPJPE 15.7 mm.
- On CMU0 with 3 cameras, AP25 improves by +10.8 over MVGFormer (cross-camera generalization).
- Cross-dataset Campus PCP average 67.3 vs 58.1 for MVGFormer, trained only on CMU Panoptic.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MV-SSM instead of MVGFormer | 3D human pose estimation · CMU Panoptic | AP25: 92.3 -> 93.5 | Helped |
| use MV-SSM instead of MVGFormer | 3D human pose estimation · Campus A1 | PCP: 40.2 -> 55.5 | Helped |
| use MV-SSM instead of MVGFormer | 3D human pose estimation · Shelf Average | PCP: 87.9 -> 88.0 | Helped |
| use MV-SSM instead of MVGFormer | cross-camera pose estimation · CMU0 (3) | AP25: 44.6 -> 55.4 | Helped |
| use MV-SSM instead of MVGFormer | cross-arrangement pose estimation · CMU3 | AP25: 52.5 -> 59.5 | Helped |
| replace PSS with a Mean block (ablation) | MV-SSM · CMU Panoptic | AP25: 93.5 -> 36.2 | Hurt |
| replace PSS with cross-attention (ablation) | MV-SSM · CMU Panoptic | AP25: 93.5 -> 90.4 | Hurt |
| remove the Mamba block (ablation) | MV-SSM · CMU Panoptic | AP25: 93.5 -> 92.3 | Hurt |
| remove GTBS and Mamba blocks (ablation) | MV-SSM · CMU Panoptic | AP25: 93.5 -> 87.7 | Hurt |
| add the PSS Kn branch instead of replacing it with an MLP (ablation) | MV-SSM · CMU Panoptic | AP25: 91.8 -> 93.5 | Helped |
| add the PSS Vn branch (ablation) | MV-SSM · CMU Panoptic | AP25: 92.5 -> 93.5 | Helped |
Surprising findings
- Removing the Mamba block only mildly hurts (AP25 92.3 vs 93.5), while removing GTBS plus Mamba causes a larger drop to 87.7.
Limitations
- Fails when a person is visible in only one camera, leading to missing keypoints.
- Occluded keypoints for missing views are sampled from a normal distribution rather than synthesized.
- Campus cross-dataset PCP remains low in absolute terms (average 67.3).
Experimental setup
Datasets: CMU Panoptic Campus Shelf
Benchmarks: CMU Panoptic Campus Shelf
Metrics: AP25 AP100 mAP MPJPE PCP
Baselines: VoxelPose MvP MVGFormer Dong et al. (geometry-based) Lin et al. Graph3D Faster VoxelPose
Builds on: Mamba VMamba (SS2D) MVGFormer MvP learnable triangulation
Closest papers by meaning
- PoseMamba: Monocular 3D Human Pose Estimation with Bidirectional Global-Local Spatio-Temporal State Space Model (AAAI 2025 · similarity 0.92)
- Sequential Joint Dependency Aware Human Pose Estimation with State Space Model (AAAI 2025 · similarity 0.90)
- PoseIRM: Enhance 3D Human Pose Estimation on Unseen Camera Settings via Invariant Risk Minimization (CVPR 2024 · similarity 0.89)
- FreeMan: Towards Benchmarking 3D Human Pose Estimation under Real-World Conditions (CVPR 2024 · similarity 0.89)
- SVTformer: Spatial-View-Temporal Transformer for Multi-View 3D Human Pose Estimation (AAAI 2025 · similarity 0.88)
- SelfPose3d: Self-Supervised Multi-Person Multi-View 3d Pose Estimation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.