Research map / Vision-language and multimodal / Human and embodied modeling
The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a multimodal T5-based language model that tokenizes speech, text and compositional body motion into one vocabulary, pre-trains with motion/audio-text alignment, and instruction-tunes for co-speech gesture generation, editable gesture generation and emotion prediction from motion.
Problem
Existing motion generation models are limited to single input modalities (speech, text, or motion), so they cannot leverage the diversity of available data or capture semantic body language such as emotion.
Approach
Represent speech, text and body-part motion tokens (face, hands, upper body, lower body) in a unified vocabulary for a Flan-T5 encoder-decoder, and pre-train with self-supervised spatial/temporal motion alignment and audio-text alignment on unpaired data before instruction-based post-training.
Key results
- On BEATv2 co-speech gesture generation, the model reaches FGD 5.301 (vs. EMAGE 5.512) and diversity 15.167 (vs. EMAGE 13.06), outperforming prior methods.
- Emotion prediction from motion achieves BLEU@1 14.71 and BERTScore 16.94, versus 1.68 and 2.31 for MotionGPT.
- With only 1/32 of paired post-training data, the full model achieves much lower FGD than the no-pre-training ablation and EMAGE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the unified multimodal language model for co-speech gesture generation | Flan-T5-Base · BEATv2 speaker-2 | FGD ×10−1: 5.512 -> 5.301 | Helped |
| Use the unified multimodal language model for co-speech gesture generation | Flan-T5-Base · BEATv2 speaker-2 | Diversity: 13.47 -> 15.167 | Helped |
| Use the unified multimodal language model for co-speech gesture generation | Flan-T5-Base · BEATv2 speaker-2 | BC ×10−1: 7.971 -> 7.780 | Hurt |
| Add language pre-training (ablation) | the co-speech gesture model · BEATv2 speaker-2 | FGD ×10−1: 7.470 -> 5.301 | Helped |
| Add multimodal pre-training (ablation) | the co-speech gesture model · BEATv2 speaker-2 | FGD ×10−1: 5.408 -> 5.301 | Helped |
| Add generative pre-training (ablation) | the co-speech gesture model · BEATv2 speaker-2 | FGD: 5.501 -> 5.301 | Helped |
| Add audio-to-text alignment (ablation) | the co-speech gesture model · BEATv2 speaker-2 | FGD: 5.443 -> 5.301 | Helped |
| Add spatial body-motion alignment (ablation) | the co-speech gesture model · BEATv2 speaker-2 | FGD: 6.336 -> 5.301 | Helped |
| Add temporal body-motion alignment (ablation) | the co-speech gesture model · BEATv2 speaker-2 | FGD: 6.800 -> 5.301 | Helped |
| Add whole-body motion alignment (ablation) | the co-speech gesture model · BEATv2 speaker-2 | FGD: 7.776 -> 5.301 | Helped |
| Use the proposed model for emotion prediction from motion | Flan-T5-Base · BEATv2 motion-to-emotion | Bleu@1: 1.68 -> 14.71 | Helped |
Surprising findings
- The model reaches strong gesture results without ever seeing speech-motion pairs during pre-training.
- MotionGPT performs at near-random level on motion-to-emotion prediction.
Limitations
- Model sometimes fails to produce coherent motion, possibly due to discrete motion tokenization.
- Relies on discrete VQ-VAE tokens rather than continuous representations.
Experimental setup
Datasets: BEATv2 LibriSpeech AMASS HumanML3D
Benchmarks: BEATv2
Metrics: FGD BC (Beat Correlation) Diversity BLEU@1 ROUGE-CIDEr BERTScore
Baselines: DisCo CaMN DiffStyleGesture Habibie et al. TalkSHOW SynTalker EMAGE MotionGPT
Builds on: T5 EMAGE MotionGPT HuBERT VQ-VAE
Closest papers by meaning
- Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs (ICLR 2025 · similarity 0.90)
- Generating Human Motion in 3D Scenes from Text Descriptions (CVPR 2024 · similarity 0.90)
- Move as You Say Interact as You Can: Language-guided Human Motion Generation with Scene Affordance (CVPR 2024 · similarity 0.89)
- AMD: Autoregressive Motion Diffusion (AAAI 2024 · similarity 0.89)
- HSI-GPT: A General-Purpose Large Scene-Motion-Language Model for Human Scene Interaction (CVPR 2025 · similarity 0.89)
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.