Research map / Large language models / Language, safety and interpretability
MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
MotionGPT fine-tunes LLaMA with LoRA to generate human motion from text plus pose keyframes, by quantizing motion into VQ-VAE codes and treating them as text tokens in an instruction prompt.
Problem
Text-to-motion generation methods support only a single control modality, limiting use with combined controls such as text plus keyframe poses, and varying motion lengths and rich motion patterns are hard to model.
Approach
Treat multimodal control signals (text and quantized pose codes from a motion VQ-VAE) as special tokens in an instruction template, and fine-tune a frozen LLaMA with LoRA using cross-entropy to output motion code sequences decoded by the VQ-VAE decoder.
Key results
- On HumanML3D text-to-motion, MotionGPT-13B reaches FID 0.567 and MM Dist 3.775, comparable to T2M-GPT (FID 0.116) and MLD (FID 0.473), with 33M trainable LoRA parameters.
- Adding a random pose keyframe condition lowers HumanML3D FID from 0.567 (text-only) to 0.367 for the 13B model.
- Joint training across text, initial, last, and key token tasks improves all metrics over separate per-task training for MotionGPT-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the LLaMA pre-training from none to 7B/13B (ablation) | MotionGPT text-to-motion · HumanML3D validation | FID: 26.01 -> 0.590 -> 0.542 | Helped |
| jointly train across tasks rather than train each task separately (ablation) | MotionGPT-7B · HumanML3D validation, text and pose-control tasks | FID: 0.670 -> $0.590^{-.180}$; 0.756 -> $0.493^{-.263}$; 1.409 -> $0.646^{-.763}$; 0.702 -> $0.390^{-.663}$ | Helped |
| add initial pose control | MotionGPT-13B · HumanML3D test | FID: 0.567 to 0.520 | Helped |
| add random keyframe pose control | MotionGPT-13B · HumanML3D test | FID: 0.567 to 0.367 | Helped |
| add last pose control | MotionGPT-13B · HumanML3D test | FID: 0.567 -> 0.591 | Hurt |
| add initial pose control | MotionGPT-13B · KIT-ML test | FID: 0.597 -> 0.664 | Hurt |
| add last pose control | MotionGPT-13B · KIT-ML test | FID: 0.597 -> 0.856 | Hurt |
| add pose control | MotionGPT-13B · HumanML3D test, initial-pose task | Recon: 24.70 -> 13.78 | Helped |
| increase LoRA rank from 8 to 32 while α=16 (ablation) | MotionGPT-7B · HumanML3D validation | FID: 0.837; 0.977; 0.576 | Mixed |
| set LoRA α to 16 with r=64 (ablation) | MotionGPT-7B · HumanML3D validation | FID: 1.869; 0.773; 0.590 | Helped |
| set training batch size to 256 rather than 128 or 512 (ablation) | MotionGPT-7B · HumanML3D validation | FID: 0.752; 0.590; 0.684 | Helped |
| replace prompt V0 with V1 or V2 (ablation) | MotionGPT-7B · HumanML3D validation | FID: 0.590; 8.506; 3.018 | Hurt |
Surprising findings
- An LLaMA randomly initialized (no pretraining) and tuned with LoRA reaches FID 26.01, far worse than pretrained LLaMA (0.590), showing LLM pretraining supplies motion priors.
- The text-only MotionGPT-13B model has worse FID than T2M-GPT on HumanML3D (0.567 vs 0.116), and MDM's KIT-ML MM Dist is much higher than the others.
Limitations
- Validated only on text and human pose modalities; other modalities such as music are not tested.
- Text-only MotionGPT lags strong baselines like T2M-GPT on FID, and pose-conditioned final-pose generation degrades slightly.
Experimental setup
Datasets: HumanML3D KIT-ML
Benchmarks: HumanML3D KIT-ML
Metrics: FID MM Dist R-Precision Diversity Recon Vel
Baselines: TEMOS TM2T T2M MotionDiffuse MDM MLD T2M-GPT
Builds on: VQ-VAE LoRA LLaMA instruction tuning T2M-GPT
Closest papers by meaning
- Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs (ICLR 2025 · similarity 0.88)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.88)
- M$^3$GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation (NeurIPS 2024 · similarity 0.88)
- ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models (AAAI 2025 · similarity 0.88)
- AvatarGPT: All-in-One Framework for Motion Understanding Planning Generation and Beyond (CVPR 2024 · similarity 0.88)
- AMD: Autoregressive Motion Diffusion (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.