Research map / Video / Video generation
OmniMotionGPT: Animal Motion Generation with Limited Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniMotionGPT, which jointly trains human and animal motion autoencoders with CLIP-based alignment to generate animal motions from text, and releases the AnimalML3D text-animal motion dataset.
Problem
Text-driven animal motion generation is hard because no paired animal text-motion dataset exists and animal skeletons differ from human ones, so human motion models cannot be directly transferred.
Approach
Jointly train motion autoencoders for human (SMPL) and animal (SMAL) skeletons on a shared primal-joint latent space, and map human latent codes to animal motion using CLIP similarity, latent consistency, and end-effector velocity losses, with human motion sampled from generators at inference.
Key results
- On out-of-distribution prompts, OMGPT reaches Top-1 R-Precision 0.850 versus 0.407 for the best baseline (MotionDiffuse).
- On the in-distribution AnimalML3D test set, OMGPT achieves Top-1 R-Precision 0.539 and FID 37.487, which is competitive with MDM and MotionDiffuse but not uniformly best.
- Ablations show removing the end-effector loss raises diversity (51.44 vs 43.80) but reduces realism, and removing the consistency loss causes incomplete motions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| transfer human motion knowledge into animal motion generation | OMGPT · AnimalML3D OOD prompts | R-Precision Top-1: 0.407±.017 -> 0.850±.009 | Helped |
| transfer human motion knowledge into animal motion generation | OMGPT · AnimalML3D OOD prompts | FID-OOD: 1.019±.014 -> 1.453±.021 | Hurt |
| transfer human motion knowledge into animal motion generation | OMGPT · AnimalML3D test set | R-Precision Top-1: 0.505±.037 -> 0.539±.064 | Helped |
| add an MLP mapping between human and animal latent spaces (ablation) | OMGPT · AnimalML3D OOD prompts | R-Precision Top-1: 0.850±.009 -> 0.351±.009 | Hurt |
| replace the linear semantic head with an MLP (ablation) | OMGPT · AnimalML3D OOD prompts | R-Precision Top-1: 0.850±.009 -> 0.404±.013 | Hurt |
| remove the end-effectors loss (ablation) | OMGPT · AnimalML3D OOD prompts | R-Precision Top-1: 0.850±.009 -> 0.477±.017 | Hurt |
| remove the latent consistency loss (ablation) | OMGPT · AnimalML3D OOD prompts | R-Precision Top-1: 0.850±.009 -> 0.508±.019 | Hurt |
Surprising findings
- On in-distribution AnimalML3D, OMGPT's FID (37.487) is worse than MDM (27.826) and MotionDiffuse (25.194), so it does not win on all metrics.
- The GPT-based baselines (T2M-GPT, MotionGPT) show high FID-OOD and diversity scores that the authors attribute to unrealistic or repetitive motions from the large codebook.
Limitations
- Training data is limited to 922 animal sequences across 23 identities, and the evaluation relies on OOD prompts without ground-truth animal motion.
- Animal motion captions are generated from DeformingThings4D meshes and skeletons were fit with Wrap4D, a commercial tool, which may limit reproducibility.
- OMGPT's in-distribution FID is worse than diffusion baselines.
Experimental setup
Datasets: HumanML3D AnimalML3D DeformingThings4D
Benchmarks: AnimalML3D test set HumanML3D out-of-distribution prompts with animal names
Metrics: R-Precision FID FID-OOD MM-Dist Diversity MModality
Baselines: T2M-GPT MotionGPT MDM MotionDiffuse
Builds on: MotionGPT T2M-GPT MDM MotionDiffuse SMAL
Closest papers by meaning
- How to Move Your Dragon: Text-to-Motion Synthesis for Large-Vocabulary Objects (ICML 2025 · similarity 0.91)
- AniMo: Species-Aware Model for Text-Driven Animal Motion Generation (CVPR 2025 · similarity 0.89)
- SnapMoGen: Human Motion Generation from Expressive Texts (NeurIPS 2025 · similarity 0.87)
- Scaling Large Motion Models with Million-Level Human Motions (ICML 2025 · similarity 0.86)
- SAMoR: Motion Modelling for Articulated Objects of Any Skeleton and Topology (NeurIPS 2026 · similarity 0.86)
- MMM: Generative Masked Motion Model (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.