Research map / 3D vision / Human pose estimation
SpatioTemporal Learning for Human Pose Estimation in Sparsely-Labeled Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STDPose, a ViT-based spatiotemporal framework that fuses multi-frame features and pose heatmaps with a Dynamic-Aware Mask and mutual-information loss to propagate and estimate poses in sparsely labeled videos.
Problem
Multi-person video pose estimation relies on dense manual annotation, and existing methods struggle with long-range temporal dependencies and with exploiting pose heatmaps alongside visual features under occlusion and blur.
Approach
Jointly models temporal visual features and temporal keypoint heatmaps via a SpatioTemporal Representation Encoder, adds a Dynamic-Aware Mask built from forward/backward heatmap residuals with a modified sigmoid, and aggregates them with cross-attention in a Pose Aggregation decoder, plus an MI-inspired loss.
Key results
- Pose propagation with T=7 reaches 90.9 mAP on PoseTrack2017 validation, 2.2 above PoseWarper.
- Pose estimation trained on full annotations achieves 87.4 mAP on PoseTrack2017, 1.0 above DiffPose.
- Pseudo-label training at T=2 (50% labeled) reaches 85.2 mAP on PoseTrack2017 validation, surpassing FAMI-Pose's 84.8; T=4 (26.7% labeled) reaches 84.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STDPose instead of PoseWarper | Video pose propagation · PoseTrack2017 Val Set | mAP: +2.2 mAP | Helped |
| Use STDPose instead of DCPose | Video pose propagation · PoseTrack2018 Val Set | mAP: 88.0 ($\uparrow$ 3.0) | Helped |
| Use STDPose instead of DCPose | Video pose propagation · PoseTrack2021 Val Set | mAP: 3.8 higher | Helped |
| Use STDPose instead of DiffPose | Video pose estimation trained on full video annotation · PoseTrack2017 Val Set | mAP: 1.0 higher | Helped |
| Use STDPose instead of TDMI | Video pose estimation trained on full video annotation · PoseTrack2018 Val Set | mAP: 0.9 mAP improvement | Helped |
| Add Temporal Feature Fusion (TFF) (ablation) | STDPose pose-propagation baseline · PoseTrack2017 Val Set | mAP: 86.8 -> 88.4; ($\uparrow$ 1.6) | Helped |
| Add Temporal Keypoints Synthesis (TKS) (ablation) | STDPose pose-propagation baseline · PoseTrack2017 Val Set | mAP: 86.8 -> 88.6; ($\uparrow$ 1.8) | Helped |
| Add TFF, TKS, and STDA (ablation) | STDPose pose-propagation baseline · PoseTrack2017 Val Set | mAP: 86.8 -> 89.6; ($\uparrow$ 2.8) | Helped |
| Add Dynamic-Aware Mask (DAM) (ablation) | STDPose with TFF, TKS, and STDA · PoseTrack2017 Val Set | mAP: 89.6 -> 90.3; performance improvement of 0.7 mAP | Helped |
| Add Mutual Information (MI) objective (ablation) | STDPose with TFF, TKS, DAM, and STDA · PoseTrack2017 Val Set | mAP: 90.3 -> 90.9; 4.1 mAP over the baseline | Helped |
| Increase time interval from T=7 to T=15, reducing labeled frames to 10.0% (ablation) | STDPose pose propagation · PoseTrack2017 Val Set | mAP: 90.9 -> 83.6 | Hurt |
| Reduce time interval to T=2, increasing labeled frames to 50.0% (ablation) | STDPose pose propagation · PoseTrack2017 Val Set | mAP: 98.1 | Helped |
Surprising findings
- Removing labeled frames (T=15, 10% labeled) causes a steep drop to 83.6 mAP for pose propagation.
Limitations
- Hyperparameters alpha and beta not densely tuned.
- Direct comparison with some recent sparse-label methods was not possible.
- Performance drops considerably at large time intervals (T=15).
Experimental setup
Datasets: PoseTrack2017 PoseTrack2018 PoseTrack2021 COCO
Benchmarks: PoseTrack2017 PoseTrack2018 PoseTrack2021
Metrics: mAP AP per joint
Baselines: PoseWarper DCPose SimpleBaseline FlowNet2 Farneback FAMI-Pose DSTA TDMI DiffPose PoseFlow
Builds on: PoseWarper DCPose ViT
Closest papers by meaning
- Video-Based Human Pose Regression via Decoupled Space-Time Aggregation (CVPR 2024 · similarity 0.87)
- MotionMap: Representing Multimodality in Human Pose Forecasting (CVPR 2025 · similarity 0.86)
- SVTformer: Spatial-View-Temporal Transformer for Multi-View 3D Human Pose Estimation (AAAI 2025 · similarity 0.86)
- SHaRPose: Sparse High-Resolution Representation for Human Pose Estimation (AAAI 2024 · similarity 0.86)
- Dynamic Camera Poses and Where to Find Them (CVPR 2025 · similarity 0.86)
- HumMUSS: Human Motion Understanding using State Space Models (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.