Research map / 3D vision / Human pose estimation
Referring Human Pose and Mask Estimation In the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces referring human pose and mask estimation (R-HPM) with text or positional prompts, a new RefHuman dataset extending COCO, and UniPHD, an end-to-end promptable transformer with a pose-centric hierarchical decoder.
Problem
Multi-person pose estimation outputs many candidates and needs selection heuristics to find a target person; existing methods lack prompt-based, joint pose-and-mask prediction for a referred individual.
Approach
A unified promptable transformer takes text or positional (scribble/point) prompts as instance queries, coordinates them with keypoint queries via deformable attention for local details and graph attention with a learnable soft adjacency matrix for global dependencies, and jointly predicts keypoints and mask of the referred person without post-processing.
Key results
- UniPHD with scribble prompts reaches 74.7 pose AP, 90.4 PCKh@0.5, and 85.7 oIoU on RefHuman val, beating Uni-GroupPose with scribble by 1.7 pose AP and 1.0 mask AP.
- UniPHD trained only with positional prompts reaches 73.9 pose AP on MS COCO val2017 at 640 resolution, above ED-Pose (68.9) and GroupPose (68.5).
- With text prompts, UniPHD achieves 66.7 pose AP and 62.2 mask AP on RefHuman val, versus 65.3 for Uni-ED-Pose.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UniPHD instead of Uni-GroupPose | RefHuman val, Scribble prompt · RefHuman val, Scribble prompt | Pose AP: 73.0 -> 74.7 (+1.7) | Helped |
| Use UniPHD instead of Uni-GroupPose | RefHuman val, Scribble prompt · RefHuman val, Scribble prompt | Mask AP: 69.0 -> 70.0 (+1.0) | Helped |
| Use scribble prompts rather than text prompts | UniPHD · RefHuman val | Pose AP: 66.7 -> 74.7 | Helped |
| Remove the pose head (ablation) | UniPHD · RefHuman val, Text prompt | Mask AP: 62.2 -> 61.8 | Hurt |
| Remove the mask head (ablation) | UniPHD · RefHuman val, Text prompt | Pose AP: 66.7 -> 63.3 | Hurt |
| Remove global dependency modeling (ablation) | UniPHD · RefHuman val, Scribble prompt | Pose AP: 74.7 -> 63.5 | Hurt |
| Remove global dependency modeling (ablation) | UniPHD · RefHuman val, Scribble prompt | Mask AP: 70.0 -> 65.8 | Hurt |
| Replace graph attention with self-attention (ablation) | UniPHD · RefHuman val, Scribble prompt | Pose AP: 74.7 -> 73.8 | Hurt |
| Replace graph attention with self-attention (ablation) | UniPHD · RefHuman val, Scribble prompt | Mask AP: 70.0 -> 69.4 | Hurt |
| Remove prompt-conditioned query initialization (ablation) | UniPHD · RefHuman val, Text prompt | Pose AP: 66.7 -> 65.5 | Hurt |
| Remove prompt-conditioned query initialization (ablation) | UniPHD · RefHuman val, Scribble prompt | Mask AP: 70.0 -> 69.6 | Hurt |
| Increase query groups from 5 to 20 (ablation) | UniPHD · RefHuman val, Text prompt | Pose AP: 62.6 -> 66.7 | Helped |
Surprising findings
- Using only a single point prompt gives advanced performance, while previous multi-person models struggle with such simple guidance in crowds.
- Without result selection, the GroupPose baseline achieves only 33.5 AP, showing the dependence on selection heuristics.
Limitations
- Text prompts underperform positional prompts due to misidentification and weak vision-language alignment.
- Human-in-the-loop GPT-4 descriptions were acceptable for only about 35% of cases, so text annotation still required heavy manual revision.
Experimental setup
Datasets: RefHuman MS COCO train2017 MS COCO val2017 RefCOCO RefCOCO+ RefCOCOg
Benchmarks: RefHuman val MS COCO val2017
Metrics: pose AP PCKh@0.5 oIoU mask AP AP_50 AP_75 AP_M AP_L FPS
Baselines: GroupPose ED-Pose PETR Uni-ED-Pose Uni-GroupPose LAVT CGFormer ReLA SgMg
Builds on: GroupPose ED-Pose PETR Swin Transformer RoBERTa
Closest papers by meaning
- PromptHMR: Promptable Human Mesh Recovery (CVPR 2025 · similarity 0.87)
- Optimizing Human Pose Estimation Through Focused Human and Joint Regions (AAAI 2025 · similarity 0.85)
- Stable-Pose: Leveraging Transformers for Pose-Guided Text-to-Image Generation (NeurIPS 2024 · similarity 0.85)
- A Large-Scale Human-Centric Benchmark for Referring Expression Comprehension in the LMM Era (NeurIPS 2024 · similarity 0.85)
- UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editing (CVPR 2025 · similarity 0.85)
- SapiensID: Foundation for Human Recognition (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.