Research map / Vision-language and multimodal / Multimodal LLMs
UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
UniPose unifies human pose comprehension, generation, and editing in one LLM (LLaVA-1.6 backbone) by tokenizing SMPL poses into a VQ-VAE codebook, adding a pose-specific ViT alongside CLIP, and using mixed causal/bidirectional attention.
Problem
Human pose comprehension, generation, and editing are studied in isolation and across single modalities; existing MLLMs lack fine-grained pose perception and a unified representation of 3D poses and text.
Approach
Treat 3D SMPL poses as a discrete language via a VQ-VAE pose tokenizer that shares a vocabulary with text, fuse CLIP and pose-estimation ViT features, and apply causal attention to text but bidirectional attention to pose tokens (with parallel pose queries) for single-step pose generation and editing.
Key results
- Text-to-Pose on PoseScript: UniPose MPJPE 308.6 and FID 0.038 vs. PoseScript's 318.0 and 0.075, with best R^T2P Top-20 89.6.
- Image-to-Text on ImageScript: UniPose R-Precision Top-1 24.5 vs. GPT4V 17.7 and LLaVA 5.7.
- Pose editing on PoseFix: UniPose MPJPE 270.3 and PA-MPJPE 138.9 vs. PoseFix 300.2 and 144.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-task training rather than training on the corresponding task only (ablation) | UniPose · Pose comprehension, generation, estimation, and editing | R-Precision Top-1 / MPJPE: 18.1 -> 85.6; 8.4 -> 67.9; 22.4 -> 24.5; 13.0 -> 13.5; 342.7 -> 308.6; 97.4 -> 94.7; 310.8 -> 270.3 | Helped |
| Add Pose-ViT to CLIP-ViT (ablation) | UniPose visual processor · Pose estimation | MPJPE: 193.4 -> 96.1 | Helped |
| Add Pose-ViT to CLIP-ViT (ablation) | UniPose visual processor · Image-to-Text | BLEU-4: 11.1 -> 13.3 | Helped |
| Use mixed attention instead of causal attention (ablation) | UniPose · Text-to-Pose | R^T2P Top-5: 9.0 -> 13.8 | Helped |
| Use mixed attention instead of causal attention (ablation) | UniPose · Text-to-Pose | Latency (s): 2.5 -> 0.2 | Helped |
| Use mixed attention instead of causal attention (ablation) | UniPose · Pose-to-Text | BLEU-4: 26.9 -> 25.0 | Hurt |
| Add global orientation noise during tokenizer training (ablation) | Pose tokenizer · AMASS | MPJPE: 6.7 -> 6.2 | Helped |
| Add global orientation noise during tokenizer training (ablation) | Pose tokenizer · MOYO | MPJPE: 32.6 -> 23.1 | Helped |
| Use UniPose instead of PoseScript | Pose-to-Text · PoseScript | R-Precision Top-1: 91.6 -> 85.6 | Hurt |
| Use UniPose instead of PoseFix | Pose-Diff · PoseFix | R-Precision Top-1: 64.6 -> 67.9 | Helped |
| Use UniPose instead of GPT4V | Image-to-Text · ImageScript | R-Precision Top-1: 17.7 -> 24.5 | Helped |
| Use UniPose instead of GPT4V | Image-Diff · ImageDiff | R-Precision Top-1: 7.3 -> 13.5 | Helped |
Surprising findings
- Mixed attention lowers Pose-to-Text BLEU-4 (25.0 vs 26.9 with causal attention) even though it improves text-to-pose retrieval.
- On 3DPW pose estimation UniPose trails specialized HMR methods (e.g., MEGA 67.5 MPJPE), which the authors attribute to the frozen visual encoder.
Limitations
- Pose estimation performance lags behind specialized HMR methods.
- Frozen visual encoder constrains pose estimation quality.
- Mixed attention slightly hurts pose-to-text linguistic metrics.
Experimental setup
Datasets: AMASS MOYO PoseScript PoseFix Human3.6M MPI-INF-3DHP COCO MPII 3DPW
Benchmarks: PoseScript PoseFix ImageScript ImageDiff 3DPW Human3.6M
Metrics: R-Precision BLEU-4 ROUGE-L METEOR MPJPE PA-MPJPE FID R@K
Baselines: PoseScript PoseFix ChatPose ChatHuman LLaVA Qwen-VL GPT-4V TokenHMR
Builds on: LLaVA VQ-VAE PoseScript PoseFix TokenHMR
Closest papers by meaning
- PoseLLaVA: Pose Centric Multimodal LLM for Fine-Grained 3D Pose Manipulation (AAAI 2025 · similarity 0.90)
- ChatPose: Chatting about 3D Human Pose (CVPR 2024 · similarity 0.89)
- UniHuman: A Unified Model For Editing Human Images in the Wild (CVPR 2024 · similarity 0.87)
- Disentangled Pose and Appearance Guidance for Multi-Pose Generation (CVPR 2025 · similarity 0.87)
- UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing (ICLR 2026 · similarity 0.86)
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.