Research map / 3D vision / Human pose estimation
Pose Priors from Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses GPT-4V contact-pair prompts converted into a differentiable vertex-distance loss to refine initial SMPL-X 3D pose estimates for two-person and self-contact images, without contact-annotated training.
Problem
3D human pose estimation from single images often fails to capture physical contact (self-contact and inter-person contact), and existing contact-aware methods require costly contact annotations or motion capture data.
Approach
Prompt a large multimodal model for pairs of touching coarse body regions, convert these into a minimum-distance loss on SMPL-X vertices, sample many LMM responses and average their losses to suppress hallucination, and jointly optimize with keypoint and pose/shape priors.
Key results
- On Hi4D, ProsePose reduces joint PA-MPJPE to 93 versus 116 for the contact heuristic and 89 for fully supervised BUDDI.
- On CHI3D, ProsePose raises average PCC to 23 versus 0 reported for BEV-level baselines and 23 vs 23 heuristic... ProsePose PCC 23 vs heuristic 23 is not reported; ProsePose improves over heuristic on PA-MPJPE (100 vs 105 on FlickrCI3D) and PCC 13 on FlickrCI3D.
- On one-person MOYO yoga poses, ProsePose improves average PCC to 25 versus 9-level HMR2 baselines, with PA-MPJPE 82 vs 84 for HMR2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LMM-predicted contact constraints to pose optimization | BEV initialization with ProsePose · Hi4D | PA-MPJPE: 116 -> 93 | Helped |
| Add LMM-predicted contact constraints to pose optimization | BEV initialization with ProsePose · FlickrCI3D | PA-MPJPE: 67 -> 58 | Helped |
| Add LMM-predicted contact constraints to pose optimization | BEV initialization with ProsePose · CHI3D | PA-MPJPE: 105 -> 100 | Helped |
| Add LMM-predicted contact constraints to pose optimization | BEV initialization with ProsePose · FlickrCI3D | Avg. PCC: 77.8 -> 79.9 | Helped |
| Add LMM-predicted contact constraints to pose optimization | BEV initialization with ProsePose · CHI3D | Avg. PCC: 74.1 -> 75.8 | Helped |
| Add LMM-predicted contact constraints to pose optimization | BEV initialization with ProsePose · MOYO | PCC @ 10 mm: 65.5 -> 73.8 | Helped |
| Add LMM contact loss to BUDDI | BUDDI · FlickrCI3D | Avg. PCC: 81.9 -> 83.2 | Helped |
| Add LMM contact loss to BUDDI | BUDDI · CHI3D | Avg. PCC: 78.0 -> 78.8 | Helped |
| Add LMM contact loss to optimization (ablation) | ProsePose · Two-person validation examples | PA-MPJPE: 81 -> 138 | Hurt |
| Add pose prior loss LGMM (ablation) | ProsePose · Two-person validation examples | PA-MPJPE: 85 -> 81 | Helped |
| Add shape regularization loss Lβ (ablation) | ProsePose · Two-person validation examples | PA-MPJPE: 91 -> 81 | Helped |
| Add initial-pose loss Lθ (ablation) | ProsePose · Two-person validation examples | PA-MPJPE: 84 -> 81 | Helped |
Surprising findings
- Identifying left/right chirality of limbs is a particular challenge for LMMs; prompting for chirality sometimes worsens results.
- Ablating the LMM loss raises two-person PA-MPJPE from 81 to 138 on FlickrCI3D validation, showing the LMM loss has the largest effect.
Limitations
- LMM hallucinations and poor left/right chirality identification limit constraint quality.
- Coarse body-region constraints limit precision compared with fine-grained contact annotations.
- Falls back to heuristic optimization frequently (e.g., 224/431 CHI3D images), which weakens gains.
Experimental setup
Datasets: Hi4D FlickrCI3D CHI3D MOYO
Benchmarks: Hi4D FlickrCI3D CHI3D MOYO
Metrics: PA-MPJPE PCC F1
Baselines: BEV Heuristic contact optimization BUDDI HMR2 HMR2+opt
Builds on: BUDDI BEV HMR2 self-consistency chain-of-thought
Closest papers by meaning
- Capturing Closely Interacted Two-Person Motions with Reaction Priors (CVPR 2024 · similarity 0.89)
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.89)
- InteractVLM: 3D Interaction Reasoning from 2D Foundational Models (CVPR 2025 · similarity 0.88)
- Reconstructing Close Human Interaction with Appearance and Proxemics Reasoning (CVPR 2025 · similarity 0.88)
- ContactField: Implicit Field Representation for Multi-Person Interaction Geometry (NeurIPS 2024 · similarity 0.88)
- ChatPose: Chatting about 3D Human Pose (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.