Research map / 3D vision / Human pose estimation
ChatPose: Chatting about 3D Human Pose
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes LLaVA-1.5 (Vicuna-13B) with a special <POSE> token and an MLP projection to output SMPL pose parameters, enabling chat-based 3D human pose estimation, text-to-pose generation, and reasoning-based pose tasks.
Problem
Existing 3D human pose estimation and generation systems are narrow and lack semantic understanding and world-knowledge reasoning; multimodal LLMs cannot yet interpret or output 3D human pose.
Approach
Treat SMPL pose as a distinct modality: the LLM emits a special <POSE> token whose hidden embedding is mapped by an MLP to SMPL 6D rotations, trained with LoRA on pose estimation and text-to-pose data plus multimodal instruction data, so world knowledge transfers to speculative and reasoning-based pose tasks.
Key results
- On the SPG benchmark, ChatPose text-to-pose recall R_T2P reaches 10.9/16.9/25.3 vs PoseScript's 2.8/4.3/7.2 (top-5/10/20).
- On H3.6M pose estimation, ChatPose gets MPJPE 126.0 mm vs LLaVA* 246.0 and GPT4-S 336.9, but trails SPIN (61.9) and HMR 2.0 (50.0).
- GPT-4-assisted evaluation gives ChatPose an overall score of 84.0 vs LLaVA-V1.5-13B's 86.5, indicating slight loss of general conversational ability.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ChatPose instead of PoseScript for speculative pose generation | PoseScript · SPG Benchmark | RP 2T retrieval recall (top 5 / 10 / 20): 1.9 / 3.8 / 6.5 -> 8.6 / 14.2 / 20.8 | Helped |
| Use ChatPose instead of PoseScript for speculative pose generation | PoseScript · SPG Benchmark | RT 2P retrieval recall (top 5 / 10 / 20): 2.8 / 4.3 / 7.2 -> 10.9 / 16.9 / 25.3 | Helped |
| Use ChatPose instead of PoseScript for text-to-pose generation | PoseScript · PoseScript test set | RT 2P retrieval recall (top 5 / 10 / 20): 22.4 / 32.1 / 43.6 -> 28.0 / 39.0 / 54.4 | Helped |
| Use ChatPose instead of PoseScript for text-to-pose generation | PoseScript · PoseScript test set | RP 2T retrieval recall (top 5 / 10 / 20): 22.6 / 31.0 / 42.3 -> 17.6 / 25.3 / 35.8 | Hurt |
| Use ChatPose instead of HMR 2.0 for image-based pose estimation | HMR 2.0 · 3DPW test set | MPJPE (mm): 91.0 -> 163.6 | Hurt |
| Use ChatPose instead of HMR 2.0 for image-based pose estimation | HMR 2.0 · Human3.6M test set | MPJPE (mm): 50.0 -> 126.0 | Hurt |
| Use ChatPose for reasoning-based pose estimation | LLaVA + PoseScript · RPE benchmark (Averaged descriptions) | MPJPE / PA-MPJPE / MPJRE (×100): 370.8 / 182.3 / 17.5 -> 307.9 / 102.9 / 12.1 | Mixed |
| Fine-tune the multimodal LLM to add ChatPose pose capabilities | LLaVA-V1.5-13B · GPT4-assisted evaluation | All: 86.5 -> 84.0 | Hurt |
| Fine-tune the multimodal LLM to add ChatPose pose capabilities | LLaVA-V1.5-13B · GPT4-assisted evaluation | Complex: 93.9 -> 96.7 | Helped |
Surprising findings
- The model shows zero-shot reasoning about poses in multi-turn dialogue even though it was never trained on multi-turn pose conversations.
- ChatPose generalizes to strong occlusions without any occlusion or blur augmentation during training.
Limitations
- 3D pose estimation accuracy remains below specialized HMR-style regressors.
- Vision encoder is frozen, limiting visual pose understanding; global orientation estimation is weak.
- Relies on noisy automatically generated pose descriptions and limited pose-language data.
Experimental setup
Datasets: PoseScript Human3.6M MPI-INF-3DHP COCO MPII LLaVA-V1.5-MIX665K 3DPW AMASS
Benchmarks: PoseScript test set SPG Benchmark RPE benchmark (3DPW-based) 3DPW Human3.6M LLaVA conversation benchmark
Metrics: retrieval recall R@5/10/20 MPJPE PA-MPJPE MPJRE
Baselines: PoseScript SPIN HMR 2.0 LLaVA GPT-4V SMPLify
Builds on: LLaVA-1.5 SMPL PoseScript LoRA HMR 2.0
Closest papers by meaning
- ChatHuman: Chatting about 3D Humans with Tools (CVPR 2025 · similarity 0.91)
- UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editing (CVPR 2025 · similarity 0.89)
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.89)
- Pose Priors from Language Models (CVPR 2025 · similarity 0.88)
- ManiPose: Manifold-Constrained Multi-Hypothesis 3D Human Pose Estimation (NeurIPS 2024 · similarity 0.86)
- Disentangled Pose and Appearance Guidance for Multi-Pose Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.