Research map / 3D vision / Human pose estimation
Optimizing Human Pose Estimation Through Focused Human and Joint Regions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VREMD, a dual-stream Transformer for video human pose estimation that uses coarse-to-fine human and keypoint masks plus a bidirectional deformable cross-attention motion module, reaching state-of-the-art mAP on PoseTrack2017/2018/2021.
Problem
Video human pose estimation is degraded by background changes, other people's movements, and weak local positional precision in Transformer-based methods, which learn motion from all pixels rather than the target body.
Approach
A two-step Human-Keypoint Mask module progressively zooms into the target person and keypoints, while a Bidirectional Motion Disentanglement module uses deformable cross attention, constrained by the enhanced features, to extract forward and backward motion cues separately.
Key results
- Achieves 87.6 mAP on PoseTrack2017 validation, 1.7 above TDMI-ST, with 84.8 wrist mAP.
- Achieves 84.6 mAP on PoseTrack2018 and 84.5 mAP on PoseTrack2021 validation sets, both state of the art.
- Ablation: HKME adds 5.1 mAP over the ViT baseline (80.2 to 85.3); BMD adds 5.4 mAP (to 85.6); combined 87.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VREMD instead of TDMI-ST | Video pose estimation · PoseTrack2017 validation | mAP: 1.7 mAP gain | Helped |
| use VREMD instead of TDMI-ST | Video pose estimation · PoseTrack2018 validation | mAP: 84.6 mAP; TDMI-ST 83.6 | Helped |
| use VREMD instead of TDMI-ST | Video pose estimation · PoseTrack2021 validation | mAP: 0.7 mAP | Helped |
| add HKME to the ViT pose baseline (ablation) | ViT pose baseline · PoseTrack2017 validation | mAP: 80.2 -> 85.3; 5.1 mAP | Helped |
| add BMD to the ViT pose baseline (ablation) | ViT pose baseline · PoseTrack2017 validation | mAP: 80.2 -> 85.6; 5.4 mAP | Helped |
| add both HKME and BMD (ablation) | ViT pose baseline · PoseTrack2017 validation | mAP: 80.2 -> 87.6 | Helped |
| add human mask without keypoint mask (ablation) | HKME · PoseTrack2017 validation | mAP: 85.9 -> 86.5 | Helped |
| add keypoint mask without human mask (ablation) | HKME · PoseTrack2017 validation | mAP: 85.9 -> 86.8 | Helped |
| use both human and keypoint masks instead of no masks (ablation) | HKME · PoseTrack2017 validation | mAP: 85.9 -> 87.6 | Helped |
| use spatio-temporal Transformers without masks instead of removing HKME (ablation) | HKME · PoseTrack2017 validation | mAP: 0.3 mAP | Helped |
| replace ADC with deformable convolution (ablation) | BMD · PoseTrack2017 validation | mAP: 0.6 mAP decrease | Hurt |
| replace ADC with deformable attention (ablation) | BMD · PoseTrack2017 validation | mAP: 85.8 | Helped |
Surprising findings
- Replacing deformable convolution with the proposed deformable cross attention plus bidirectional separation gave gains, but the paper notes that plain deformable convolution underperformed deformable attention by 1.1 mAP.
Limitations
- Head joint localization is suboptimal due to interference from nearby joints such as the shoulder.
- Under severe occlusion by other people, the model may incorporate temporal cues from non-target individuals.
Experimental setup
Datasets: PoseTrack2017 PoseTrack2018 PoseTrack2021 COCO
Benchmarks: PoseTrack2017 PoseTrack2018 PoseTrack2021
Metrics: mAP per-joint AP
Baselines: DCPose TDMI-ST FAMI-Pose DSTA HANet M-HANet KPM DetTrack PoseWarper MDPN
Builds on: DCPose TDMI deformable convolution deformable attention ViT
Closest papers by meaning
- Detect-and-Track: Efficient Pose Estimation in Videos (CVPR 2018 · similarity 0.90)
- End-to-End Multi-Person Pose Estimation with Pose-Aware Video Transformer (AAAI 2026 · similarity 0.90)
- PoseFormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose Estimation (CVPR 2023 · similarity 0.90)
- Temporal Feature Alignment and Mutual Information Maximization for Video-Based Human Pose Estimation (CVPR 2022 · similarity 0.90)
- Attention Mechanism Exploits Temporal Contexts: Real-Time 3D Human Pose Reconstruction (CVPR 2020 · similarity 0.90)
- Video-Based Human Pose Regression via Decoupled Space-Time Aggregation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.