Research map / Vision-language and multimodal / Human and embodied modeling
PoseLLaVA: Pose Centric Multimodal LLM for Fine-Grained 3D Pose Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PoseLLaVA, a LLaVA-based multimodal LLM with SMPL pose encoder-decoder that performs language-guided pose estimation, generation, and adjustment, plus the PosePart body-part-level adjustment dataset.
Problem
Fine-grained natural-language 3D human pose manipulation (estimation, generation, adjustment) is underexplored; existing methods focus on coarse motion commands and lack fine-grained pose-language data.
Approach
Integrates SMPL 6D pose tokens as a third modality into LLaVA via a pose encoder-decoder, aligned to the vision encoder with multi-scale contrastive and regression losses, then trained in three stages; also builds PosePart, a single-body-part pose adjustment dataset.
Key results
- Pose adjustment on PoseFix: MPJPE 71.7 vs 130.5 for PoseFix and 180.5 for ChatPose.
- Pose generation: MPJPE 169.72 vs 214.3 for PoseScript, a reduction of roughly 20%.
- Pose estimation on Human3.6M: MPJPE 71.7, worse than SPIN (60.2) but better than ChatPose (177.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PoseLLaVA instead of ChatPose for pose estimation | PoseLLaVA · Human3.6M | MPJPE: 177.8 -> 71.7 | Helped |
| use PoseLLaVA instead of ChatPose for pose estimation | PoseLLaVA · Human3.6M | PA-MPJPE: 55.7 -> 51.2 | Helped |
| use PoseLLaVA instead of SPIN for pose estimation | PoseLLaVA · Human3.6M | MPJPE: 60.2 -> 71.7 | Hurt |
| use PoseLLaVA instead of SPIN for pose estimation | PoseLLaVA · 3DPW | PA-MPJPE: 52.3 -> 81.0 | Hurt |
| use PoseLLaVA-Pose instead of PoseFix | PoseLLaVA-Pose · PoseFix Dataset | MPJPE: 130.5 -> 71.7 | Helped |
| use PoseLLaVA-Pose instead of PoseFix | PoseLLaVA-Pose · PoseFix Dataset | R@1: 25% -> 37% | Helped |
| use PoseLLaVA instead of PoseScript | PoseLLaVA · PoseScript | MPJPE: 214.3 -> 169.72 | Helped |
| add PosePart training data to PoseFix training data (ablation) | PoseLLaVA · PoseFix Dataset | MPJPE: 74.6 -> 71.7 | Helped |
| add PoseFix training data to PosePart training data (ablation) | PoseLLaVA · PosePart Dataset | MPJPE: 62.9 -> 61.6 | Helped |
| train on PosePart only instead of PoseFix only (ablation) | PoseLLaVA · PoseFix Dataset | MPJPE: 74.6 -> 83.5 | Hurt |
Surprising findings
- Pose estimation on 3DPW has higher PA-MPJPE (81.0) than ChatPose (79.0) in Table 2, so the claimed comparable performance is not clearly supported.
- PosePart-only training gives worse MPJPE on PoseFix than PoseFix-only training, showing limited cross-dataset transfer.
Limitations
- Dataset focuses on single-person poses with simple backgrounds.
- Pose estimation accuracy lags traditional task-specific methods such as SPIN and HRM2.0.
- Fine-grained pose generation accuracy remains limited.
Experimental setup
Datasets: Human3.6M PoseScript PoseFix PosePart AMASS 3DPW
Benchmarks: Human3.6M 3DPW PoseFix PosePart PoseScript
Metrics: MPJPE PA-MPJPE R@1 R@5 R@10
Baselines: SPIN HRM2.0 ChatPose PoseFix ControlNet PoseScript
Builds on: LLaVA ChatPose PoseFix PoseScript LISA
Closest papers by meaning
- UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editing (CVPR 2025 · similarity 0.90)
- Pose Priors from Language Models (CVPR 2025 · similarity 0.87)
- Stable-Pose: Leveraging Transformers for Pose-Guided Text-to-Image Generation (NeurIPS 2024 · similarity 0.86)
- ChatPose: Chatting about 3D Human Pose (CVPR 2024 · similarity 0.86)
- VersaCamVLA: Camera-Configurable VLA Policies for Robotic Manipulation (NeurIPS 2026 · similarity 0.86)
- PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.