Research map / Video / Video generation
Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Motion-Agent pairs GPT-4 as a conversational planner with MotionLLM, a LoRA-adapted Gemma2-2B that translates between text and discrete motion tokens, to support multi-turn motion generation, editing, and captioning.
Problem
3D human motion generation and understanding methods typically need extensive task-specific training, handle long complex prompts poorly, and lack multi-turn conversational control.
Approach
Motion is quantized into discrete tokens added to a pre-trained LLM's vocabulary; only LoRA adapters are trained on text-motion pairs, and a frozen GPT-4 decomposes user requests into multiple MotionLLM calls whose token outputs are concatenated and detokenized, enabling long and edited motions without extra training.
Key results
- MotionLLM reaches R-precision Top-1 0.515 and FID 0.230 on HumanML3D generation, competitive with trained-from-scratch models while fine-tuning only a small fraction of parameters.
- On HumanML3D captioning, MotionLLM reports BLEU@4 17.65, ROUGE 48.7, and BERTScore 42.63, above TM2T, MotionGPT, and MotionChain.
- MotionLLM reports the best Diversity (9.908) among the compared generation methods on HumanML3D.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MotionLLM instead of MoMask for text-to-motion generation | MotionLLM · HumanML3D test set | Diversity: 9.620^{ \pm .064} -> 9.908^{ \pm .102} | Helped |
| Use MotionLLM instead of MoMask for text-to-motion generation | MotionLLM · HumanML3D test set | FID: 0.045^{ \pm .002} -> 0.230^{ \pm .009} | Hurt |
| Use MotionLLM instead of MoMask for text-to-motion generation | MotionLLM · HumanML3D test set | R Precision Top 1: 0.521^{ \pm .002} -> 0.515^{ \pm .004} | Hurt |
| Use MotionLLM instead of prior captioning models | MotionLLM · HumanML3D test set | Bleu@4: 12.56 -> 17.65 | Helped |
| Use MotionLLM instead of prior captioning models | MotionLLM · HumanML3D test set | Cider: 33.70 -> 33.74 | Helped |
| Increase LoRA rank from 16 to 64 (ablation) | MotionLLM with Gemma2-2b · KIT-ML | R Precision Top 1: 0.411 -> 0.422 | Helped |
| Increase LoRA rank from 16 to 64 (ablation) | MotionLLM with Gemma2-2b · KIT-ML | FID: 0.745 -> 0.658 | Helped |
| Increase LoRA rank from 16 to 64 (ablation) | MotionLLM with Gemma2-2b · KIT-ML | Multimodal Dist: 2.994 -> 2.929 | Helped |
| Increase LoRA rank from 16 to 64 (ablation) | MotionLLM with Gemma2-2b · KIT-ML | Diversity: 11.313 -> 11.195 | Hurt |
| Use Gemma2-9b instead of Gemma2-2b at LoRA rank 32 (ablation) | MotionLLM · KIT-ML | R Precision Top 1: 0.415 -> 0.439 | Helped |
| Use Gemma2-9b instead of Gemma2-2b at LoRA rank 32 (ablation) | MotionLLM · KIT-ML | FID: 0.712 -> 0.438 | Helped |
| Use Gemma2-9b instead of Gemma2-2b at LoRA rank 32 (ablation) | MotionLLM · KIT-ML | Diversity: 11.251 -> 11.151 | Hurt |
Surprising findings
- Smaller open LLMs (Llama-3-7B, Mixtral-8x7B) often failed to follow the required JSON format, so the agent could not parse their plans.
Limitations
- No 3D scene or environment interaction modeling, and no detailed hand or facial motion.
- Multi-human generation is only a preliminary trial and not fully explored.
- Long-motion and multi-turn results are qualitative, with no established ground truth or quantitative evaluation.
Experimental setup
Datasets: HumanML3D KIT-ML
Benchmarks: HumanML3D KIT-ML
Metrics: R-precision Top-1 R-precision Top-3 FID MultiModal Distance Diversity BLEU@1 BLEU@4 ROUGE CIDEr BERTScore
Baselines: T2M TM2T MDM MLD MotionDiffuse T2M-GPT MotionGPT MotionChain MoMask
Builds on: T2M-GPT MotionGPT LoRA VQ-VAE
Closest papers by meaning
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.90)
- ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models (AAAI 2025 · similarity 0.88)
- MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators (AAAI 2024 · similarity 0.88)
- Generating Human Motion in 3D Scenes from Text Descriptions (CVPR 2024 · similarity 0.88)
- AvatarGPT: All-in-One Framework for Motion Understanding Planning Generation and Beyond (CVPR 2024 · similarity 0.88)
- HSI-GPT: A General-Purpose Large Scene-Motion-Language Model for Human Scene Interaction (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.