Research map / 3D vision / Human pose estimation
PromptHMR: Promptable Human Mesh Recovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PromptHMR, a transformer that estimates SMPL-X human meshes from full images conditioned on spatial prompts (boxes, face boxes, masks) and semantic prompts (CLIP shape descriptions, interaction labels), plus a video variant with temporal layers and TRAM-style world grounding.
Problem
Human pose and shape estimation struggles in crowded scenes, close person-person interactions, and monocular body shape estimation; cropped-image methods discard scene context while full-image methods miss people and are less accurate.
Approach
A promptable HPS regressor that encodes full-image DINOv2 features together with box, mask, text-shape and interaction prompts, with random prompt masking in training, plus promptable cross-person attention layers for interacting pairs.
Key results
- On RICH, PromptHMR* reaches 37.3 PA-MPJPE / 65.5 PVE, and on CHI3D it reaches 58.5 Pair-PA-MPJPE versus BUDDI's 68 (PA-MPJPE 27.2 vs 47 for BUDDI).
- On EMDB world-grounded motion, PromptHMR-Vid achieves WA-MPJPE100 71.0 and W-MPJPE100 216.5, beating TRAM (76.4 / 222.4).
- Text shape prompts reduce HBW height/chest/waist/hip errors from 69/51/88/63 mm to 62/43/76/58 mm when used at train and test time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add shape descriptions during training and testing (ablation) | PromptHMR · HBW | Height: 69 -> 62 | Helped |
| Add shape descriptions during training and testing (ablation) | PromptHMR · HBW | Chest: 51 -> 43 | Helped |
| Add shape descriptions during training and testing (ablation) | PromptHMR · HBW | Waist: 88 -> 76 | Helped |
| Add shape descriptions during training and testing (ablation) | PromptHMR · HBW | Hip: 63 -> 58 | Helped |
| Add shape descriptions during training and testing (ablation) | PromptHMR · P2P-20k | P2P-20k: 26 -> 24 | Helped |
| Add shape descriptions during training only (ablation) | PromptHMR · HBW | Chest: 51 -> 48 | Helped |
| Add shape descriptions during training only (ablation) | PromptHMR · HBW | Height: 69 -> 69 | No clear effect |
| Add mask prompts (ablation) | PromptHMR · HI4D | Pair-PA-MPJPE: 87.2 -> 83.0 | Helped |
| Add the interaction module (ablation) | PromptHMR · HI4D | Pair-PA-MPJPE: 87.2 -> 73.0 | Helped |
| Train on HI4D (ablation) | PromptHMR · HI4D | Pair-PA-MPJPE: 87.2 -> 52.6 | Helped |
| Add mask prompts and the interaction module, and train on HI4D (ablation) | PromptHMR · HI4D | Pair-PA-MPJPE: 87.2 -> 47.9 | Helped |
| Add the interaction module when trained on HI4D (ablation) | PromptHMR · HI4D | PA-MPJPE: 36.3 -> 36.5 | No clear effect |
Surprising findings
- Training with shape descriptions alone improves shape accuracy even when no text prompt is given at test time.
Limitations
- Shape descriptions and interaction prompts must be supplied by the user rather than generated automatically by VLMs.
- As a monocular regression method it still produces interpenetration between closely interacting people.
- Other side information types (action descriptions, scene context, body measurements) are not explored.
Experimental setup
Datasets: BEDLAM AGORA 3DPW COCO MPII AIC InstaVariety CHI3D HI4D HBW P2P-20k
Benchmarks: 3DPW EMDB RICH HI4D CHI3D HBW
Metrics: MPJPE PA-MPJPE PVE Pair-PA-MPJPE WA-MPJPE100 W-MPJPE100 RTE Jitter Foot Skating
Baselines: CLIFF HMR2.0 TokenHMR CameraHMR BEV Multi-HMR WHAM TRAM GVHMR BUDDI
Builds on: SMPL-X DINOv2 SAM prompt decoder design TRAM SHAPY
Closest papers by meaning
- Referring Human Pose and Mask Estimation In the Wild (NeurIPS 2024 · similarity 0.87)
- GenHMR: Generative Human Mesh Recovery (AAAI 2025 · similarity 0.86)
- FinePOSE: Fine-Grained Prompt-Driven 3D Human Pose Estimation via Diffusion Models (CVPR 2024 · similarity 0.86)
- TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation (CVPR 2024 · similarity 0.85)
- AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimation (CVPR 2024 · similarity 0.85)
- SAT-HMR: Real-Time Multi-Person 3D Mesh Estimation via Scale-Adaptive Tokens (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.