Research map / Video / Video generation
HumanTOMATO: Text-aligned Whole-body Motion Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HumanTOMATO, a text-driven whole-body (face, hand, body) motion generation framework using a holistic hierarchical VQ-VAE, hierarchical GPT, facial cVAE, and a TMR text-motion retrieval prior for alignment.
Problem
Text-driven generation of whole-body human motion (body, hands, face) that is vivid and well aligned with text at sequence level; existing methods are body-only and use frame-level CLIP/LLM alignment.
Approach
Hierarchical VQ-VAE with separate body and hand codebooks (body quantizer conditioned on hand tokens), a Hierarchical-GPT predicting body then hand codes conditioned on a TMR text embedding, plus a sequence-level contrastive text-motion alignment loss from a pre-trained TMR model; facial motion via a text-conditioned cVAE.
Key results
- On Motion-X, HumanTOMATO achieves FID 1.174 vs. 1.366 for T2M-GPT and 3.407 for MLD, and TMR-R-Precision Top1 0.416 vs. 0.385 for MLD.
- H2VQ reduces whole-body reconstruction MPJPE on Motion-X to 92.966 mm vs. 110.94 for RVQ and 139.33 for vanilla VQ (1024).
- Using TMR text embedding plus TMR alignment loss improves FID-related and alignment metrics over CLIP embedding (Matching-score 1.732 vs. baseline numbers in table; TMR-Matching-score 3.894 vs. 3.917 CLIP).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use H²VQ instead of RVQ for hierarchical body-hand quantization (ablation) | whole-body motion reconstruction · Motion-X | MPJPE (mm), All: 110.94 -> 92.966 | Helped |
| use H²VQ instead of RVQ for hierarchical body-hand quantization (ablation) | whole-body motion reconstruction · GRAB | MPJPE (mm), All: 62.938 -> 46.742 | Helped |
| increase Vanilla VQ codebook size from 512 to 1024 (ablation) | Vanilla VQ reconstruction · Motion-X | MPJPE (mm), All: 140.66 -> 139.33 | Helped |
| replace TMR text embedding with CLIP embedding (ablation) | HumanTOMATO · Motion-X | TMR-R-Precision (256) Top1: 0.416 -> 0.405 | Hurt |
| add TMR text-motion alignment supervision (ablation) | HumanTOMATO with TMR embedding · Motion-X | TMR-R-Precision (256) Top1: 0.416 -> 0.416 | Helped |
| add TMR text-motion alignment supervision (ablation) | HumanTOMATO with TMR embedding · Motion-X | TMR-R-Precision (256) Top2: 0.596 -> 0.603 | Helped |
| remove rotation information from the motion representation (ablation) | VAE reconstruction · HumanML3D | MPJPE: 49.51 -> 46.84 | Helped |
| remove rotation and velocity information from the motion representation (ablation) | VAE reconstruction · HumanML3D | MPJPE: 49.51 -> 51.43 | Hurt |
| remove rotation information from the motion representation (ablation) | VQ-VAE reconstruction · HumanML3D | MPJPE: 78.24 -> 68.86 | Helped |
Surprising findings
- Simply increasing the codebook size of vanilla VQ-VAE yields almost no reconstruction improvement for whole-body motion.
- The original R-Precision retriever is less accurate than the proposed TMR model, and its retrieval gap grows at larger candidate set sizes.
Limitations
- Whole-body motion data is scarce; relies on Motion-X as the only large whole-body text-motion dataset.
- Facial motion is generated by a separate cVAE and is only loosely coupled with body and hand generation.
Experimental setup
Datasets: Motion-X GRAB HumanML3D
Benchmarks: Motion-X HumanML3D GRAB
Metrics: FID TMR-R-Precision (256) R-Precision (32) TMR-Matching-score Matching-score MModality Diversity MPJPE
Baselines: T2M-GPT MDM MLD MotionCLIP RVQ Vanilla VQ-VAE
Builds on: VQ-VAE T2M-GPT TEMOS TMR residual vector quantization
Closest papers by meaning
- BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics (CVPR 2024 · similarity 0.90)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.88)
- Expressive Forecasting of 3D Whole-Body Human Motions (AAAI 2024 · similarity 0.88)
- SemGeoMo: Dynamic Contextual Human Motion Generation with Semantic and Geometric Guidance (CVPR 2025 · similarity 0.88)
- Shape My Moves: Text-Driven Shape-Aware Synthesis of Human Motions (CVPR 2025 · similarity 0.87)
- InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.