Research map / 3D vision / 3D generation
How to Move Your Dragon: Text-to-Motion Synthesis for Large-Vocabulary Objects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a text-to-motion diffusion framework for arbitrary skeletal rigs by annotating the Truebones Zoo animal motion dataset with multi-level captions, adding rig augmentation, and extending motion diffusion transformers with tree and rest-pose encodings.
Problem
Text-driven 3D motion synthesis for large-vocabulary objects is hard due to a lack of annotated, diverse-skeleton motion data and because existing motion diffusion models assume a fixed skeleton template.
Approach
Makes a motion diffusion transformer skeleton-agnostic by tokenizing per joint and adding tree positional encoding (TreePE) and rest-pose encoding (RestPE), combined with rig augmentation (bone length, joint number, rest pose resetting) and a two-stage pose-then-motion training with factorized spatial-temporal attention.
Key results
- Full model (TreePE+RestPE+rig aug) reaches 0.6802 FID and 0.6045 R@1 on the test set vs 2.2564 FID and 0.2626 R@1 without positional encodings or rig augmentation (pose level).
- In a 30-participant user study, the full method is preferred 65.60% of the time vs 12.40% for w/o rig augmentation and 2.27% for GPT-4o captions.
- Retargeting achieves the highest MSI (9.93e-3) while the proposed method scores 8.78e-3, the highest among data-driven methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use human-annotated captions instead of GPT-generated captions (ablation) | Text-to-motion model · User study | Preference (%): 2.27 -> 65.60 | Helped |
| Add rig augmentation (ablation) | Text-to-motion model · User study | Preference (%): 12.40 -> 65.60 | Helped |
| Use shared cross-object learning instead of single-object motion models | Text-to-motion model · User study | Preference (%): 10.67 -> 65.60 | Helped |
| Use learned text-conditioned synthesis instead of retargeting | Text-to-motion model · User study | Preference (%): 9.07 -> 65.60 | Helped |
| Add TreePE and RestPE (ablation) | Pose diffusion model · Pose diffusion Test Set | R@1: 0.2626 -> 0.5979 | Helped |
| Add rig augmentation (ablation) | Pose diffusion model with TreePE and RestPE · Pose diffusion Test Set | R@1: 0.5979 -> 0.6045 | Helped |
| Add rig augmentation (ablation) | Pose diffusion model without TreePE and RestPE · Pose diffusion Test Set | R@1: 0.2626 -> 0.2562 | Hurt |
| Add TreePE and RestPE (ablation) | Pose diffusion model · Pose diffusion Test Set+ | R@1: 0.3705 -> 0.4435 | Helped |
| Add rig augmentation (ablation) | Pose diffusion model with TreePE and RestPE · Pose diffusion Test Set+ | R@1: 0.4435 -> 0.6707 | Helped |
| Add rig augmentation (ablation) | Pose diffusion model without TreePE and RestPE · Pose diffusion Test Set+ | R@1: 0.3705 -> 0.3867 | Helped |
| Use human-annotated captions instead of GPT-generated captions (ablation) | Text-to-motion model · Varied rig styles | Coverage AUC | Helped |
| Add rig augmentation (ablation) | Text-to-motion model · Varied rig styles | Coverage AUC | Helped |
Surprising findings
- Captions generated by GPT-4o led to much lower user preference (2.27%) than human-annotated captions, showing caption quality matters a lot.
- Models without positional encodings give near-chance retrieval on the test set (R@1 0.26), indicating that rig information is essential for generalization.
Limitations
- Uses only joint-wise Euler rotations and ignores global root translation.
- Augmented skeletons can violate physical plausibility constraints such as symmetry and balance.
- Does not generalize well to human or open-vocabulary objects in zero-shot settings due to the animal-only training data.
Experimental setup
Datasets: Truebones Zoo
Benchmarks: Truebones Zoo held-out split external Sketchfab objects
Metrics: FID R-Precision Alignment Coverage Multimodality Motion Stability Index user preference
Baselines: Single-Object Motion Diffusion Models Retargeting w/o Rig Augmentation w/ GPT-4o captions
Builds on: motion diffusion model (MDM) SiT adaLN-Zero SigLIP Truebones Zoo
Closest papers by meaning
- OmniMotionGPT: Animal Motion Generation with Limited Data (CVPR 2024 · similarity 0.91)
- AniMo: Species-Aware Model for Text-Driven Animal Motion Generation (CVPR 2025 · similarity 0.88)
- SAMoR: Motion Modelling for Articulated Objects of Any Skeleton and Topology (NeurIPS 2026 · similarity 0.88)
- Single Motion Diffusion (ICLR 2024 · similarity 0.88)
- Scaling Large Motion Models with Million-Level Human Motions (ICML 2025 · similarity 0.87)
- AMD: Anatomical Motion Diffusion with Interpretable Motion Decomposition and Fusion (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.