Research map / Large language models / Agents, code and math
AvatarGPT: All-in-One Framework for Motion Understanding Planning Generation and Beyond
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AvatarGPT, an all-in-one LLM framework (T5-based) that jointly trains seven motion tasks (understanding, planning, decomposition, generation, in-between, scene estimation, summarization) with discrete motion tokens and an automatically annotated in-the-wild video dataset.
Problem
Human motion tasks (generation, understanding, planning, decomposition) are handled by siloed models, and long-motion synthesis from high-level user instructions requires end-to-end planning and generation.
Approach
Treats each motion-related task as an instruction for a shared LLM, with motion encoded as VQ-VAE tokens mapped into an extended vocabulary through a lightweight adapter and a separate motion LM head, plus an unsupervised video-to-text annotation pipeline to build high-level planning data; the tasks form a closed loop enabling unlimited long-motion synthesis.
Key results
- On motion understanding (HumanML3D), Ours achieves BLEU-4 12.70 and CIDEr 32.65 vs MotionGPT's 12.47 and 29.20, and R-Precision Top-1 0.510 on generation vs MotionGPT 0.492.
- On high-level tasks, T5-Large reaches LCS up to 0.994 (T2S) and 0.997 (S2T), outperforming GPT2-Large on 7 of 8 subtasks.
- Ablation: adding the vocabulary adapter improves T5-Large generation R-Precision Top-1 from 0.497 to 0.510 and FID from 0.215 to 0.168.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use T5-Large instead of GPT2-Large (ablation) | AvatarGPT · HumanML3D motion generation | R Top-1: 0.454 -> 0.510 | Helped |
| Use T5-Large instead of GPT2-Large (ablation) | AvatarGPT · HumanML3D motion understanding | Blue-4: 13.29 -> 12.70 | Hurt |
| Use T5-Large instead of GPT2-Large (ablation) | AvatarGPT · High-level tasks | LCS: 7 of 8 tasks | Mixed |
| Use T5-Large instead of T5-Base (ablation) | AvatarGPT · HumanML3D motion generation | R Top-1: 0.468 -> 0.510 | Helped |
| Add the vocabulary adapter instead of extending the LLM vocabulary and learning weights from scratch (ablation) | AvatarGPT · HumanML3D motion generation | R Top-1: 0.497 -> 0.510 | Helped |
| Add the vocabulary adapter instead of extending the LLM vocabulary and learning weights from scratch (ablation) | AvatarGPT · HumanML3D motion generation | FID: 0.215 -> 0.168 | Helped |
| Add the vocabulary adapter instead of extending the LLM vocabulary and learning weights from scratch (ablation) | AvatarGPT · HumanML3D motion understanding | BertScore: 49.34 -> 53.58 | Helped |
| Fine-tune Llama-13B with LoRA to connect motion modality with text (ablation) | AvatarGPT · HumanML3D motion generation | R Top-1: 0.510 -> 0.389 | Hurt |
Surprising findings
- LoRA-tuned Llama-13B performs poorly on generation and understanding, attributed to LoRA training under 1% of parameters.
- GPT2-Large slightly outperforms T5-Large on motion understanding BLEU-1 (though T5-Large wins on most other tasks).
Limitations
- High-level tasks are evaluated with ChatGPT-based LCS and small user studies rather than ground truth.
- No ground truth exists for full-pipeline long-motion evaluation; cycle consistency is used instead.
- Low-level results are mixed; e.g., motion-in-between and FID are not uniformly best.
Experimental setup
Datasets: HumanML3D in-the-wild videos annotated via automatic pipeline
Benchmarks: HumanML3D LCS high-level benchmark (8 subtasks)
Metrics: R-Precision FID Diversity MultiModality BLEU-1 BLEU-4 ROUGE-L CIDEr BERTScore Logical Coherence Score
Baselines: TM2T MDM MLD T2M-GPT MotionGPT HMD
Builds on: VQ-VAE (T2M-GPT) T5 InstructGPT Gato LoRA
Closest papers by meaning
- PlanLLM: Video Procedure Planning with Refinable Large Language Models (AAAI 2025 · similarity 0.89)
- UniMo: Unified Motion Generation and Understanding with Chain of Thought (AAAI 2026 · similarity 0.88)
- Video Language Planning (ICLR 2024 · similarity 0.88)
- Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs (ICLR 2025 · similarity 0.88)
- MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators (AAAI 2024 · similarity 0.88)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.