Research map / 3D vision / Human pose estimation
Video-Based Human Pose Regression via Decoupled Space-Time Aggregation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DSTA, a regression-based video human pose estimator that decouples spatial (between adjacent joints) and temporal (per-joint trajectory) attention over joint-wise feature tokens, avoiding heatmaps.
Problem
Heatmap-based multi-frame pose estimators are computationally and storage heavy per frame, limiting real-time and edge deployment, while regression-based pose estimators built for static images degrade badly on video input (occlusion, motion blur, defocus).
Approach
Convert backbone features into one feature token per joint, then apply joint-wise local-awareness attention separately for temporal (same joint across frames) and spatial (joints within structural groups) dependencies, concatenating the two before regressing coordinates with RLE loss.
Key results
- On PoseTrack2017 val, DSTA with HRNet-W48 reaches 84.6 mAP, 8.9 above image-based RLE (75.7), and 83.4 mAP with 0.02 GFLOPs head vs DCPose's 82.8 mAP at 11.0 GFLOPs.
- With ViT-H backbone DSTA achieves 85.6 mAP on PoseTrack2017 val.
- At 64x64 input resolution DSTA outperforms DCPose by 20.3 mAP (55.4 vs 35.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use video-based DSTA instead of image-based Poseur | PoseTrack2017 pose regression · PoseTrack2017 val. | mAP: 79.3 -> 84.6 | Helped |
| Use video-based DSTA instead of DCPose | HRNet-W48 video pose estimation · PoseTrack2017 val. | mAP: 82.8 -> 84.6 | Helped |
| Add DSTA's low-computation regression head instead of DCPose's heatmap head | HRNet-W48 video pose estimation · PoseTrack2017 val. | GFLOPs of head: 11.0 -> 0.02 | Helped |
| Add JFD and STD to represent and aggregate joint-level features (ablation) | DSTA · PoseTrack2017 val. | mAP: 73.8 -> 78.6 | Helped |
| Model spatiotemporal relations jointly rather than decoupling space and time (ablation) | DSTA · PoseTrack2017 val. | mAP: 78.6 -> 74.8 | Hurt |
| Add temporal dependency modeling (TD) (ablation) | DSTA with spatial dependency modeling · PoseTrack2017 val. | mAP: 71.4 -> 78.6 | Helped |
| Add spatial dependency modeling (SD) (ablation) | DSTA with temporal dependency modeling · PoseTrack2017 val. | mAP: 78.1 -> 78.6 | Helped |
| Use the proposed JFD instead of [23] method (b) for joint embeddings (ablation) | DSTA · PoseTrack2017 val. | mAP: 77.9 -> 78.6 | Helped |
| Use more auxiliary frames (ablation) | DSTA · PoseTrack2017 val. | mAP: 78.0 -> 78.6 -> 79.7; 82.6 -> 83.4 -> 84.6; 82.6 -> 84.3 -> 85.6 | Helped |
| Use DSTA rather than DCPose at low input resolutions (ablation) | Video pose estimation · PoseTrack2017 val. | mAP: DSTA: 77.9; DCPose: 71.7; DSTA: 55.4; DCPose: 35.1 | Helped |
Surprising findings
- Temporal modeling per joint (TD module) contributes much more than spatial modeling (SD module): 78.1 vs 71.4 mAP when used alone.
Limitations
- Pose-regression head still relies on the top-down paradigm with an external person detector and cropped clips.
- Spatial dependency modeling (SD) contributes relatively little, since joint tokens already carry some spatial context.
- Evaluated only on PoseTrack datasets for video pose estimation.
Experimental setup
Datasets: COCO PoseTrack2017 PoseTrack2018 PoseTrack21
Benchmarks: PoseTrack2017 PoseTrack2018 PoseTrack21
Metrics: mAP GFLOPs parameters
Baselines: RLE Poseur DCPose PoseWarper FAMI-Pose DetTrack HRNet MDPN
Builds on: Poseur RLE PoseWarper DCPose Transformer
Closest papers by meaning
- Optimizing Human Pose Estimation Through Focused Human and Joint Regions (AAAI 2025 · similarity 0.88)
- SVTformer: Spatial-View-Temporal Transformer for Multi-View 3D Human Pose Estimation (AAAI 2025 · similarity 0.88)
- SpatioTemporal Learning for Human Pose Estimation in Sparsely-Labeled Videos (AAAI 2025 · similarity 0.87)
- PoseMamba: Monocular 3D Human Pose Estimation with Bidirectional Global-Local Spatio-Temporal State Space Model (AAAI 2025 · similarity 0.87)
- Sequential Joint Dependency Aware Human Pose Estimation with State Space Model (AAAI 2025 · similarity 0.87)
- MotionMap: Representing Multimodality in Human Pose Forecasting (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.