Research map / Vision-language and multimodal / Human and embodied modeling
MG-MotionLLM: A Unified Framework for Motion Comprehension and Generation across Multiple Granularities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MG-MotionLLM, a T5-based unified motion-language model that handles coarse and fine-grained (body-part, temporal) motion comprehension and generation via a multi-granularity pretraining scheme with 28 tasks followed by task-specific instruction tuning.
Problem
Existing motion-language LLMs model only coarse whole-sequence text, so they cannot handle fine-grained, body-part- and time-specific motion understanding, generation, and editing.
Approach
Discretize motions with a T2M-GPT VQ-VAE, extend the T5 vocabulary with motion tokens, and pretrain on 28 coarse and fine-grained tasks (including temporal localization of snippets from detailed body-part scripts) before task-specific instruction tuning, so that granularities mutually reinforce each other.
Key results
- Text-to-motion on HumanML3D: Top-3 R-Precision 0.802 and FID 0.303 for the base model, vs. MotionGPT's 0.778 and 0.232 (MotionGPT has lower FID but lower R-precision).
- Motion-to-text on HumanML3D: Bleu@4 of 8.06 and BERTScore 36.7 vs. MotionGPT's 12.47 and 32.4, with R-Precision Top-1 0.592 vs. 0.543.
- Granularity-synergy pretraining plus instruction tuning improves (Text + Detailed Text)-to-Motion Top-3 from 0.750 (direct tuning) to 0.779 (full scheme).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Granularity-Synergy pretraining before task-specific instruction tuning (ablation) | MG-MotionLLM · HumanML3D, Text-to-Motion | Top-3: 0.773 -> 0.802 | Helped |
| add task-specific instruction tuning after Coarse + Fine pretraining (ablation) | MG-MotionLLM · HumanML3D, Text-to-Motion | Top-3: 0.767 -> 0.802 | Helped |
| include fine-grained tasks alongside coarse-grained tasks in pretraining (ablation) | MG-MotionLLM · HumanML3D, Text-to-Motion | Top-3: 0.725 -> 0.767 | Helped |
| include coarse-grained tasks alongside fine-grained tasks in pretraining (ablation) | MG-MotionLLM · HumanML3D, (Text + Detailed Text)-to-Motion | Top-3: 0.718 -> 0.771 | Helped |
| add detailed descriptions to coarse text as motion-generation conditioning | MG-MotionLLM · HumanML3D, motion generation | Top-3: 0.773 -> 0.750 | Hurt |
| increase model size from T5-Small to T5-Base (ablation) | MG-MotionLLM · HumanML3D, Text-to-Motion | Top-3: 0.781 -> 0.802 | Helped |
| increase model size from T5-Base to T5-Large (ablation) | MG-MotionLLM · HumanML3D, Text-to-Motion | Top-3: 0.802 -> 0.782 | Hurt |
| increase model size from T5-Base to T5-Large (ablation) | MG-MotionLLM · FineMotion test, snippet-level motion-to-detailed-text | BLEU@4: 46.94 -> 49.26 | Helped |
Surprising findings
- Directly instruction-tuning on coarse plus detailed text lowered Top-3 retrieval (0.750) relative to coarse-only text (0.773), which the authors attribute to the length of detailed descriptions.
- Scaling T5 from base to large did not consistently improve results, which the authors attribute to the limited size of HumanML3D.
Limitations
- Scaling to larger T5 models did not improve performance, limited by the small HumanML3D dataset.
- Fine-grained tasks rely on FineMotion, which re-labels HumanML3D with detailed descriptions, and the editing and fine-grained applications are shown only qualitatively.
Experimental setup
Datasets: HumanML3D FineMotion
Benchmarks: HumanML3D FineMotion
Metrics: FID R-Precision MM-Dist Diversity MModality BLEU ROUGE BERTScore
Baselines: TEMOS TM2T MotionDiffuse T2M-GPT FineMoGen MoMask MotionGPT
Builds on: T2M-GPT MotionGPT T5 HumanML3D
Closest papers by meaning
- M$^3$GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation (NeurIPS 2024 · similarity 0.91)
- ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models (AAAI 2025 · similarity 0.88)
- HSI-GPT2: A Dual-Granularity Large Motion Reasoning Model with Diffusion Refinement for Human-Scene Interaction (CVPR 2026 · similarity 0.87)
- FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understanding (NeurIPS 2025 · similarity 0.87)
- Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs (ICLR 2025 · similarity 0.87)
- OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.