Research map / 3D vision / Human pose estimation
HiPART: Hierarchical Pose AutoRegressive Transformer for Occluded 3D Human Pose Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates hierarchical dense 2D poses from sparse 2D keypoints via a VQ-VAE-style multi-scale skeletal tokenizer and a center-to-periphery, sparse-to-dense autoregressive transformer, then lifts them to 3D with a vanilla spatial transformer.
Problem
Occluded 3D human pose estimation from monocular images is hard because the sparse 2D skeleton input (e.g., 17 joints) carries too little local context for 2D-to-3D lifting.
Approach
Densify the sparse 2D pose into hierarchical dense 2D poses (48 and 96 joints) using a multi-scale VQ-VAE tokenizer with skeleton-aware alignment, and generate the tokens autoregressively in a center-to-periphery, sparse-to-dense order rather than next-token order.
Key results
- On Human3.6M with CPN 2D inputs, HiPART reaches 42.0 mm MPJPE in the single-frame setting, outperforming the multi-frame MixSTE (f=81) at 42.4 mm.
- Hierarchical 2D pose input improves the vanilla spatial transformer from 52.1 mm to 42.0 mm MPJPE on Human3.6M.
- On 3DPW-AdvOcc@80, the method reaches 143.2 mm MPJPE versus 153.6 mm for Di2Pose.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| generate hierarchical dense 2D poses from sparse 2D input | HiPART · Human3.6M, CPN detected 2D poses | MPJPE: 42.4 -> 42.0 | Helped |
| generate hierarchical dense 2D poses from sparse 2D input | HiPART · 3DPW | MPJPE: 79.3 -> 77.2 | Helped |
| generate hierarchical dense 2D poses from sparse 2D input | HiPART · 3DPW-AdvOcc@40 | MPJPE: 108.4 -> 124.6 | Hurt |
| generate hierarchical dense 2D poses from sparse 2D input | HiPART · 3DPW-AdvOcc@80 | MPJPE: 153.6 -> 143.2 | Helped |
| add hierarchical codebooks instead of one shared codebook (ablation) | HiPART · Human3.6M | MPJPE: 45.3 -> 42.0 | Helped |
| add Local Alignment (ablation) | HiPART · Human3.6M | MPJPE: 43.6 -> 42.0 | Helped |
| add Global Alignment (ablation) | HiPART · Human3.6M | MPJPE: 42.4 -> 42.0 | Helped |
| replace classification with autoregressive modeling (ablation) | HiPART · Human3.6M | MPJPE: 46.3 -> 43.6 | Helped |
| use center-to-periphery generation (ablation) | HiPART autoregressive modeling · Human3.6M | MPJPE: 43.6 -> 42.5 | Helped |
| use sparse-to-dense generation (ablation) | HiPART autoregressive modeling · Human3.6M | MPJPE: 43.6 -> 43.1 | Helped |
| combine center-to-periphery and sparse-to-dense generation (ablation) | HiPART autoregressive modeling · Human3.6M | MPJPE: 43.6 -> 42.0 | Helped |
| add HiPART hierarchical pose inputs (ablation) | vanilla spatial transformer · Human3.6M | MPJPE: 52.1 -> 42.0 | Helped |
Surprising findings
- Adding a 192-joint pose level gives only 0.6 mm gain over 48+96 joints despite higher complexity.
Limitations
- Training and evaluation for the hierarchical dense 2D ground truth depend on the 3D mesh of Human3.6M, which is unavailable in real-world settings.
- On 3DPW-AdvOcc@40 the method is worse than Di2Pose in MPJPE (124.6 vs 108.4 mm).
Experimental setup
Datasets: Human3.6M 3DPW 3DPW-Occ 3DPW-AdvOcc DTD
Benchmarks: Human3.6M 3DPW 3DPW-Occ 3DPW-AdvOcc
Metrics: MPJPE PA-MPJPE FPS parameters
Baselines: DiffPose Di2Pose PCT MixSTE PoseFormer MHFormer VideoPose Pose2Mesh HGN Graformer
Builds on: VQ-VAE-2 PCT MLP-Mixer Pose2Mesh
Closest papers by meaning
- HiPoser: 3D Human Pose Estimation with Hierarchical Shared Learning at Parts-Level Using Inertial Measurement Units (AAAI 2025 · similarity 0.89)
- $\text{Di}^2\text{Pose}$: Discrete Diffusion Model for Occluded 3D Human Pose Estimation (NeurIPS 2024 · similarity 0.88)
- Disentangled Diffusion-Based 3D Human Pose Estimation with Hierarchical Spatial and Temporal Denoiser (AAAI 2024 · similarity 0.87)
- SVTformer: Spatial-View-Temporal Transformer for Multi-View 3D Human Pose Estimation (AAAI 2025 · similarity 0.86)
- PandaPose: 3D Human Pose Lifting from a Single Image via Propagating 2D Pose Prior to 3D Anchor Space (NeurIPS 2025 · similarity 0.86)
- Lifting by Image – Leveraging Image Cues for Accurate 3D Human Pose Estimation (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.