Research map / 3D vision / Human pose estimation
SelfPose3d: Self-Supervised Multi-Person Multi-View 3d Pose Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SelfPose3d, a self-supervised multi-view multi-person 3D pose estimator that trains VoxelPose-style networks using only multi-view images and HRNet 2D pseudo poses, via synthetic 3D root localization, cross-affine-view differentiable heatmap rendering, and adaptive supervision attention.
Problem
Multi-person multi-view 3D human pose estimation normally requires 3D ground-truth poses (from dense camera systems) for training; the paper asks whether this dependence can be removed using only calibrated multi-view images and noisy 2D pseudo poses.
Approach
Learning-by-projection with a VoxelPose backbone: 3D root localization is trained on synthetic random 3D root points rendered into views plus affine-consistency, and the bottleneck 3D poses are projected to all views under two affine augmentations and rendered as differentiable heatmaps supervised by pseudo 2D poses, with hard (L1) and soft (L2) attention to downweight noisy pseudo labels.
Key results
- On Panoptic, SelfPose3d reaches 96.4 AP50 and 24.5 mm MPJPE versus 98.3 AP50 and 17.7 mm for fully-supervised VoxelPose.
- On Shelf and Campus (trained on pseudo 3D poses from Panoptic), SelfPose3d averages 95.1 and 87.9 PCP respectively, vs 97.0 and 96.7 for fully-supervised VoxelPose.
- Ablation: using HRNet pseudo 2D poses with L1+L2 losses and attention gives 24.5 mm MPJPE; ground-truth 2D poses lower it to 19.9 mm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ground-truth 2d poses instead of pseudo 2d poses (ablation) | SelfPose3d · Panoptic | MPJPE: 24.5 -> 19.9 | Helped |
| add affine augmentations and cross-affine-view consistency (ablation) | SelfPose3d · Panoptic | AP50: 86.0 -> 93.8 | Helped |
| add affine augmentations and cross-affine-view consistency (ablation) | SelfPose3d · Panoptic | MPJPE: 34.7 -> 29.3 | Helped |
| add affine augmentations without cross-affine-view consistency (ablation) | SelfPose3d · Panoptic | AP50: 86.0 -> 83.3 | Hurt |
| add L1 pose loss to L2 pose loss (ablation) | SelfPose3d · Panoptic | MPJPE: 25.7 -> 24.5 | Helped |
| use L1 pose loss only (ablation) | SelfPose3d · Panoptic | training convergence | Hurt |
| add L1 loss attention (ablation) | SelfPose3d · Panoptic | MPJPE: 28.5 -> 26.3 | Helped |
| add L2 loss attention (ablation) | SelfPose3d · Panoptic | MPJPE: 28.5 -> 25.0 | Helped |
| add both L1 and L2 loss attention (ablation) | SelfPose3d · Panoptic | MPJPE: 28.5 -> 24.5 | Helped |
| replace HRNet pseudo-pose generator with Keypoint R-CNN (R-101) (ablation) | SelfPose3d · Panoptic | MPJPE: 29.3 -> 31.9 | Hurt |
Surprising findings
- Using only L1 joint loss without the L2 heatmap loss fails to converge due to label noise.
- On Campus, SelfPose3d (87.9 average PCP) lags fully-supervised VoxelPose (96.7) noticeably.
Limitations
- Gap to fully-supervised VoxelPose on Panoptic (96.4 vs 98.3 AP50; 24.5 vs 17.7 MPJPE).
- Campus performance markedly below fully-supervised methods (87.9 vs 96.7 average PCP).
- Depends on off-the-shelf 2D pseudo poses whose occlusion errors require adaptive attention; used only 9 of the Panoptic videos for training.
Experimental setup
Datasets: Panoptic Shelf Campus COCO
Benchmarks: Panoptic Shelf Campus
Metrics: AP25 AP50 AP100 AP150 Recall@500 MPJPE PCP
Baselines: VoxelPose MvP Lin et al. Wu et al. TEMPO MvPose Pirinen et al. (ACTOR) 3DPS Ershadi et al.
Builds on: VoxelPose HRNet Mask R-CNN learning-by-projection
Closest papers by meaning
- CanonPose: Self-Supervised Monocular 3D Human Pose Estimation in the Wild (CVPR 2021 · similarity 0.90)
- Self-Supervised 3D Human Pose Estimation via Part Guided Novel Image Synthesis (CVPR 2020 · similarity 0.90)
- Geometry-Driven Self-Supervised Method for 3D Human Pose Estimation (AAAI 2020 · similarity 0.90)
- DisPOSE: Projected Polystochastic Diffusion for Self-Supervised Multi-View 3D Human Pose Estimation (ICML 2026 · similarity 0.89)
- MV-SSM: Multi-View State Space Modeling for 3D Human Pose Estimation (CVPR 2025 · similarity 0.88)
- Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.