Research map / 3D vision / Human pose estimation
Lifting by Image – Leveraging Image Cues for Accurate 3D Human Pose Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a two-stage image-assisted lifting framework for monocular 3D human pose estimation that uses a Pose-guided Transformer Layer and an Adaptive Feature Selection Module to focus keypoints on informative image patches and prune background tokens.
Problem
Lifting-based 3D human pose estimation suffers from depth ambiguity, and methods that add image features generalize poorly because training data is narrow and attention concentrates on dataset-biased background.
Approach
A Pose-guided Transformer Layer uses the keypoint-to-image attention matrix (normalized before transposition) so that image tokens receive updates weighted by importance, suppressing background; an adaptive module then prunes low-attention image tokens and a second stage refines keypoints using only retained features.
Key results
- On Human3.6M under Protocol 1 with CPN 2D input, the method reaches 46.4mm average MPJPE among single-frame-plus-image methods, outperforming listed single-frame baselines such as diffPose (49.7mm).
- Trained on H3.6M and tested without fine-tuning on MPI-INF-3DHP, the method achieves 88.2 PCK, 59.3 AUC and 68.9mm MPJPE, the best of the compared methods.
- Pose-guided Transformer Layer reduces background attention from 73% to 60% on H3.6M and improves MPJPE from 30.4mm to 29.2mm (coarse pose).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Stage 2 focused refinement after the coarse pose (ablation) | Progressive learning framework · Human3.6M and MPI-INF-3DHP | MPJPE: 29.2 -> 28.6; 70.9 -> 68.9 | Helped |
| Use end-to-end training with fine supervision only instead of progressive learning (ablation) | Progressive learning framework · Human3.6M and MPI-INF-3DHP | MPJPE: 28.6 -> 32.6; 68.9 -> 76.4 | Hurt |
| Use end-to-end training with both supervisions instead of progressive learning (ablation) | Progressive learning framework · Human3.6M and MPI-INF-3DHP | MPJPE: 28.6 -> 30.2; 68.9 -> 72.4 | Hurt |
| Add the Pose-guided Transformer Layer (ablation) | Broad Query stage · Human3.6M and MPI-INF-3DHP | MPJPE: 30.4 -> 29.2; 74.2 -> 70.9 | Helped |
| Add the Pose-guided Transformer Layer (ablation) | Broad Query stage · Human3.6M and MPI-INF-3DHP | Background Attention: 73% -> 60%; 75% -> 63% | Helped |
| Use 2 standard Transformer Layers and remove the Pose-guided Layer (ablation) | Stage 1 · Human3.6M | MPJPE: 29.2 -> 31.3 | Hurt |
| Set the image-token retention rate to 0.01 (ablation) | Adaptive Feature Selection Module · Human3.6M | MPJPE: 28.6 -> 28.9 | Hurt |
| Set the image-token retention rate to 1, retaining all tokens (ablation) | Adaptive Feature Selection Module · Human3.6M | MPJPE: 28.6 -> 29.0 | Hurt |
| Pretrain the image encoder on MS COCO 2D human pose estimation rather than ImageNet classification (ablation) | HRNet-w32 image encoder · Human3.6M | MPJPE: 35.2 -> 28.6 | Helped |
| Use HRNet-w32 instead of HRNet-w48 (ablation) | Image encoder · Human3.6M | MPJPE: 30.3 -> 28.6 | Helped |
| Normalize the attention map before transposition rather than after (ablation) | Pose-guided Transformer Layer · Human3.6M and MPI-INF-3DHP | MPJPE: 29.6 -> 29.2; 80.9 -> 70.9 | Helped |
| Replace normalization before transposition with double normalization (ablation) | Pose-guided Transformer Layer · Human3.6M and MPI-INF-3DHP | MPJPE: 29.2 -> 29.8; 70.9 -> 71.1 | Hurt |
Surprising findings
- Image-based models trained on H3.6M lose accuracy on MPI-INF-3DHP (MPJPE 74.2mm), indicating that image features can hurt cross-dataset generalization.
- Keypoints attend to structurally related body regions (e.g., elbow for wrist) rather than only their own image location.
Limitations
- Generalization evaluated only by training on H3.6M and testing on MPI-INF-3DHP with ground-truth 2D input.
- Retention rate and layer configuration are tuned via ablations on the test benchmarks.
Experimental setup
Datasets: Human3.6M MS COCO 2017
Benchmarks: Human3.6M MPI-INF-3DHP
Metrics: MPJPE P-MPJPE PCK AUC
Baselines: Graformer Monocular SemGCN Learning MHFormer VideoPose diffPose Pose-Oriented MGCN GraphSH Simple Cascaded
Builds on: Graformer HRNet Transformer
Closest papers by meaning
- PandaPose: 3D Human Pose Lifting from a Single Image via Propagating 2D Pose Prior to 3D Anchor Space (NeurIPS 2025 · similarity 0.92)
- PoseFormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose Estimation (CVPR 2023 · similarity 0.89)
- Geometry-Driven Self-Supervised Method for 3D Human Pose Estimation (AAAI 2020 · similarity 0.89)
- A Single 2D Pose with Context is Worth Hundreds for 3D Human Pose Estimation (NeurIPS 2023 · similarity 0.88)
- Lifting Motion to the 3D World via 2D Diffusion (CVPR 2025 · similarity 0.88)
- Deep Semantic Graph Transformer for Multi-View 3D Human Pose Estimation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.