Research map / 3D vision / Human pose estimation
Causal-Inspired Multitask Learning for Video-Based Human Pose Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CM-Pose, a causal-inspired multitask framework for video pose estimation that adds masked-token reconstruction and token-denoising auxiliary tasks and then selects causal tokens and clusters non-causal tokens, reaching state-of-the-art mAP on PoseTrack2017/2018/2021.
Problem
Video-based human pose estimation struggles with challenging scenes such as pose occlusion and video defocus, and existing methods lack causal reasoning and interpretability about which tokens matter for keypoints.
Approach
Add masked-token reconstruction and token-denoising as auxiliary tasks sharing a criss-cross spatio-temporal attention network (training-only), then at inference separate causal (keypoint-attended) tokens from non-causal tokens, cluster the non-causal tokens with DPC-KNN and merge them by importance-weighted sum before heatmap prediction.
Key results
- CM-Pose reaches 87.5 mAP on PoseTrack2017, a 1.8 mAP gain over TDMI (85.7).
- CM-Pose reaches 84.4 mAP on PoseTrack2018, versus 83.5 for TDMI and DSTA.
- CM-Pose reaches 84.3 mAP on PoseTrack2021, versus 83.5 for TDMI and DSTA; ablation shows removing both auxiliary tasks drops PoseTrack2017 mAP from 87.5 to 85.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add masked token reconstruction task (ablation) | CM-Pose · PoseTrack2017 val | Mean: 87.0 -> 87.5 | Helped |
| add denoising token task (ablation) | CM-Pose · PoseTrack2017 val | Mean: 86.9 -> 87.5 | Helped |
| add both masked token reconstruction and denoising tasks (ablation) | CM-Pose · PoseTrack2017 val | Mean: 85.6 -> 87.5 | Helped |
| retain non-causal tokens alongside causal tokens (ablation) | CM-Pose · PoseTrack2017 val | Mean: 85.4 -> 87.5 | Helped |
| add Fine Token Enhancement (ablation) | CM-Pose · PoseTrack2017 val | Mean: 86.0 -> 87.5 | Helped |
| use CM-Pose instead of TDMI | video-based pose estimation · PoseTrack2017 val | mAP: 85.7 -> 87.5 | Helped |
| use CM-Pose instead of TDMI | video-based pose estimation · PoseTrack2018 val | mAP: 83.5 -> 84.4 | Helped |
| use CM-Pose instead of DSTA | video-based pose estimation · PoseTrack2021 val | mAP: 83.5 -> 84.3 | Helped |
Surprising findings
- Keeping non-causal tokens helps: using only causal tokens drops mAP from 87.5 to 85.4, and removing the whole FTE module gives 86.0.
Limitations
- Gains on PoseTrack2018 and 2021 are reported on validation sets with a fixed set of hyperparameters (masking and noise ratios tuned on PoseTrack2017 validation).
- Interpretability claims rest mainly on qualitative visualizations and keypoint-attention selection rather than quantitative causal analysis.
- Limited description of the criss-cross attention design and of how the causal framing is formally justified.
Experimental setup
Datasets: PoseTrack2017 PoseTrack2018 PoseTrack2021
Benchmarks: PoseTrack2017 PoseTrack2018 PoseTrack2021
Metrics: mAP AP per keypoint
Baselines: HRNet CorrTrack Dynamic-GNN PoseWarper DCPose SLT-Pose HANet M-HANet FAMI-Pose DSTA TDMI
Builds on: ViT TokenPose DPC-KNN TDMI DCPose
Closest papers by meaning
- Dual Coding Theory in Action: Language-Assisted Human Pose Estimation in Videos (AAAI 2026 · similarity 0.87)
- Temporal Feature Alignment and Mutual Information Maximization for Video-Based Human Pose Estimation (CVPR 2022 · similarity 0.87)
- Video-Based Human Pose Regression via Decoupled Space-Time Aggregation (CVPR 2024 · similarity 0.85)
- From Prompts to Tokens: Internalizing Causal Supervision in Vision-Language Model for Multi-Image Causal Reasoning (ICML 2026 · similarity 0.85)
- Causality-Inspired Invariant Representation Learning for Text-Based Person Retrieval (AAAI 2024 · similarity 0.84)
- Optimizing Human Pose Estimation Through Focused Human and Joint Regions (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.