Research map / Vision-language and multimodal / Human and embodied modeling
M$^3$GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M$^3$GPT, a unified LLM (T5-base) framework that tokenizes motion, music and text into one vocabulary, jointly fine-tunes the motion de-tokenizer with the LLM, and uses text as a bridge via auxiliary music-to-text and text-to-dance tasks to handle six motion comprehension/generation tasks.
Problem
Existing motion-language models handle only one non-motion modality or only a subset of motion tasks, and motion comprehension and generation are studied in isolation; integrating text, music and motion into one framework with bidirectional alignment is hard.
Approach
Discretize motion, dance and music into a shared token vocabulary for a T5 backbone, jointly optimize the LLM with the motion de-tokenizer using raw-space L1 reconstruction loss, and use text as a bridge through auxiliary music-to-text and text-to-dance tasks to create synergy between music-to-dance and text-to-motion.
Key results
- On AIST++ music-to-dance, M$^3$GPT (instruction-tuned) reaches FID_k 24.34 versus 75.47 for the trained single-task model; the auxiliary tasks cut FID_k by nearly 10 points.
- On Motion-X motion-to-text, M$^3$GPT (instruction-tuned) reports CIDEr 42.93 versus 31.36 for MotionGPT, but text-to-motion R TOP1 0.615 trails MoMask's 0.668.
- On Motion-X motion in-between, M$^3$GPT (pre-trained) attains FID 0.604 versus 0.831 for MDM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the T2D and A2T auxiliary tasks (ablation) | M³GPT multitask training · Motion-X text-to-motion | R TOP1: 0.547 -> 0.615 | Helped |
| Add the T2D and A2T auxiliary tasks (ablation) | M³GPT multitask training · AIST++ music-to-dance | FID_k: 37.14 -> 24.34 | Helped |
| Jointly optimize the motion de-tokenizer with the LLM (ablation) | M³GPT · AIST++ music-to-dance | FID_k: 28.46 -> 24.34 | Helped |
| Jointly optimize the motion de-tokenizer with the LLM (ablation) | M³GPT · Motion-X text-to-motion | FID: 0.091 -> 0.093 | Hurt |
| Instruction-tune the multitask-pretrained model only on text-to-motion (ablation) | M³GPT · Motion-X text-to-motion | R TOP1: 0.656 -> 0.661 | Helped |
| Instruction-tune the multitask-pretrained model only on music-to-dance (ablation) | M³GPT · FineDance music-to-dance | FID_k: 86.47 -> 42.66 | Helped |
Surprising findings
- Joint optimization with the motion de-tokenizer slightly increases text-to-motion FID (under 0.003), attributed to simple text-driven motion patterns.
- Naive multitask joint training yields inferior per-task performance versus single-task training, motivating the text-bridge design.
Limitations
- Models only body motion, excluding hands and faces.
- Music-to-dance and dance-to-music evaluated on small datasets (AIST++, FineDance).
- Joint de-tokenizer optimization slightly raises text-to-motion FID.
Experimental setup
Datasets: Motion-X AIST++ FineDance
Benchmarks: Motion-X AIST++ FineDance
Metrics: R TOP1 FID Diversity BLEU@4 CIDEr FID_k Div_k BAS BCS BHS
Baselines: MLD T2M-GPT MotionDiffuse TM2T MDM MoMask MotionGPT Bailando
Builds on: MotionGPT VQ-VAE Jukebox T5
Closest papers by meaning
- MG-MotionLLM: A Unified Framework for Motion Comprehension and Generation across Multiple Granularities (CVPR 2025 · similarity 0.91)
- ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models (AAAI 2025 · similarity 0.89)
- MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators (AAAI 2024 · similarity 0.88)
- HSI-GPT: A General-Purpose Large Scene-Motion-Language Model for Human Scene Interaction (CVPR 2025 · similarity 0.88)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.87)
- Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.