Research map / 3D vision / Human pose estimation
Cross-view and Cross-pose Completion for 3D Human Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pre-trains a ViT encoder-decoder with cross-view and cross-pose completion on human body and hand image pairs, then fine-tunes it for human mesh recovery, DensePose, and hand gesture/grasp tasks.
Problem
Standard ImageNet pre-training for human-centric vision suffers from domain shift and object-centric bias, while collecting 2D/3D human labels does not scale.
Approach
Generalizes CroCo's cross-view masked image modeling to dynamic pairs (cross-pose completion) of different poses from human motion, combined with cross-view pairs from multi-view data, and a human-centric masking strategy that prioritizes visible human patches.
Key results
- CroCo-Body reaches 44.2 PA-MPJPE and 73.3 MPJPE on 3DPW after fine-tuning on Mix*, state of the art among model-free methods listed (vs. FastMETRO 44.6 / 73.5).
- On DensePose COCO, CroCo-Body improves AUC10 from 0.378 to 0.416 and AUC30 from 0.614 to 0.692 over DensePose.
- CroCo-Hand† (pre-trained on more diverse hand-object data) reaches 49.9 Best View and 49.6 All on GUN-71, versus 22.7/20.5 for Rogez et al.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CroCo-Body pre-training instead of FastMETRO | body mesh recovery · 3DPW test (Mix*) | PA-MPJPE: 44.6 -> 44.2 | Helped |
| use CroCo-Body rather than the DensePose baseline | DensePose estimation · COCO DensePose | AUC30: 0.614 -> 0.692 | Helped |
| use CroCo-Hand rather than METRO | hand mesh recovery · FreiHand | PA-MPJPE: 6.8 -> 7.4 | Hurt |
| pre-train CroCo-Hand with more diverse hand-object interaction data | CroCo-Hand · GUN-71 (Best View) | accuracy (%): 37.8 -> 49.9 | Helped |
| use CroCo-Hand pre-training instead of MAE-Hand | hand gesture classification · HaGRID (64 samples/class) | accuracy (%): 95.9 -> 97.4 | Helped |
| initialize CroCo-Body pre-training from CroCo weights rather than random weights (ablation) | CroCo-Body · 3DPW test | PA-MPJPE: 60.5 -> 57.5 | Helped |
| initialize CroCo-Hand pre-training from CroCo weights rather than random weights (ablation) | CroCo-Hand · FreiHand | PA-MPJPE: 8.63 -> 8.21 | Helped |
| focus masking on human patches rather than masking uniformly (ablation) | CroCo-Body · COCO DensePose | AUC30: 0.654 -> 0.692 | Helped |
| add 2D keypoint supervision to CroCo-Body pre-training (ablation) | CroCo-Body · 3DPW test | PA-MPJPE: 53.8 -> 52.8 | Helped |
| use binocular instead of monocular input (ablation) | CroCo-Body · 3DPW test | PA-MPJPE: 57.5 -> 55.0 | Helped |
| use binocular instead of monocular input (ablation) | CroCo-Hand · HanCo | PA-MPJPE: 11.0 -> 9.5 | Helped |
| use CroCo-Body pre-training rather than MAE-IN1k | body mesh recovery · 3DPW test | PA-MPJPE: 59.6 -> 57.4 | Helped |
Surprising findings
- MAE pre-trained on the same human-centric data (MAE-Body/Hand) performs worse than MAE pre-trained on ImageNet-1k, suggesting gains come from the pretext objective rather than data.
- Pre-training with 2D keypoints supervision is outperformed by the self-supervised CroCo-Body pre-training on all downstream tasks.
Limitations
- Grasp classification does not reach state of the art with the default pre-training data due to limited background variety and lack of manipulated objects.
- Cross-pose data depends on monocular videos, and multi-view data diversity is limited; segmentation for human masking requires an off-the-shelf parser (PGN).
- Binocular hand and body results rely on a fixed set of fine-tuning datasets.
Experimental setup
Datasets: HUMBI AIST SURREAL-style synthetic data COCO AMASS 3DPW PoseTrack PennAction JRDB MARS InterHand2.6M DexYCB
Benchmarks: 3DPW DensePose COCO FreiHand HaGRID GUN-71 HanCo
Metrics: PA-MPJPE MPJPE MPVPE AUC accuracy
Baselines: CroCo MAE-IN1k MAE-Body/Hand Random initialization 2D keypoints supervised pre-training HybrIK PARE METRO MeshGraphormer FastMETRO PointHMR DensePose
Builds on: CroCo MAE DPT SURREAL
Closest papers by meaning
- Unsupervised Learning of Category-Level 3D Pose from Object-Centric Videos (CVPR 2024 · similarity 0.87)
- HUNTER: Unsupervised Human-centric 3D Detection via Transferring Knowledge from Synthetic Instances to Real Scenes (CVPR 2024 · similarity 0.87)
- Self-Supervised 3D Human Mesh Recovery from a Single Image with Uncertainty-Aware Learning (AAAI 2024 · similarity 0.86)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.86)
- Common3D: Self-Supervised Learning of 3D Morphable Models for Common Objects in Neural Feature Space (CVPR 2025 · similarity 0.85)
- ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.