Research map / Video / Video generation
MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controls
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MotionCraft, a two-stage coarse-to-fine diffusion transformer with MC-Attn (static and dynamic topology graphs) and plug-and-play control branches for whole-body text-, speech-, and music-driven motion generation, plus the MC-Bench SMPL-X benchmark.
Problem
Unified whole-body motion generation conditioned on text, speech, or music suffers from motion distribution drift across tasks, optimization conflicts when mixing conditions of different granularities, and inconsistent motion formats across datasets.
Approach
Pretrain a text-to-motion diffusion transformer, then freeze it and attach copied control branches for speech or music via zero-initialized bridges; the MC-Attn module models a learnable static skeleton graph in parallel with an input-dependent dynamic topology graph and temporal attention.
Key results
- On the MC-Bench HumanML3D subset, MotionCraft-Mix reaches FID 6.707 and R-Precision Top-1 0.600, best among compared methods (FineMoGen FID 7.323).
- On BEAT2 speech-to-gesture, MotionCraft-Mix achieves FID_H 12.882 and FID_B 25.187, best among listed baselines (MCM FID_H 23.946).
- On FineDance music-to-dance, MotionCraft-Mix achieves FID_H 2.849 and FID_B 67.159, versus MCM FID_H 4.717 and FID_B 78.577.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the text-to-motion backbone on all MC-Bench data rather than the HumanML3D subset | MotionCraft · HumanML3D (MC-Bench) | FID: 8.477 -> 6.707 | Helped |
| Train the text-to-motion backbone on all MC-Bench data rather than the HumanML3D subset | MotionCraft · BEAT2 (Speech-to-Gesture) | FID_H: 18.486 -> 12.882 | Helped |
| Train the text-to-motion backbone on all MC-Bench data rather than the HumanML3D subset | MotionCraft · FineDance (Music-to-Dance) | FID_H: 3.858 -> 2.849 | Helped |
| Train the text-to-motion backbone on all MC-Bench data rather than the HumanML3D subset | MotionCraft · FineDance (Music-to-Dance) | Div: 16.667 -> 18.483 | Helped |
| Add static topology modeling to dynamic topology modeling (ablation) | MotionCraft · HumanML3D (MC-Bench) | FID: 8.455 -> 6.707 | Helped |
| Add dynamic topology modeling to static topology modeling (ablation) | MotionCraft · BEAT2 (Speech-to-Gesture) | FID_H: 12.929 -> 12.882 | Helped |
| Add static topology modeling to dynamic topology modeling (ablation) | MotionCraft · FineDance (Music-to-Dance) | FID_H: 4.972 -> 2.849 | Helped |
| Increase model size from 77M to 130M parameters (ablation) | MotionCraft-Mix · HumanML3D (MC-Bench) | FID: 6.707 -> 5.593 | Helped |
| Increase model size from 130M to 478M parameters (ablation) | MotionCraft-Mix · HumanML3D (MC-Bench) | FID: 5.593 -> 7.872 | Hurt |
| Fully unfreeze the body-wise encoder and decoder rather than locally unfreezing task-relevant parts (ablation) | MotionCraft · BEAT2 (Speech-to-Gesture) | FID_H: 17.962 -> 15.346 | Helped |
| Use temporal patching rather than modeling each frame as a unit (ablation) | MotionCraft · BEAT2 (Speech-to-Gesture) | FID_H: 17.583 -> 15.346 | Helped |
Surprising findings
- Scaling the model from 77M to 478M parameters yields a rise-then-fall trend, which the authors attribute to limited high-quality data.
- Patching adjacent frames into one temporal unit hurts motion performance, contrary to general time-series practice.
Limitations
- Relies on sequence-level pseudo-captions for speech and music datasets lacking text.
- Axis-angle SMPL-X representation causes root rotation and trajectory fluctuations.
- Performance declines when model scale grows without more high-quality data.
Experimental setup
Datasets: HumanML3D BEAT2 FineDance AMASS AIST++
Benchmarks: MC-Bench HumanML3D subset original HumanML3D BEAT2 FineDance
Metrics: R-Precision FID Diversity MM Dist FID_H FID_B Face L2 Beat Align Score
Baselines: T2M-GPT MDM MotionDiffuse FineMoGen MCM Motion-Verse TalkShow EMAGE Edge FineDance
Builds on: MDM ControlNet ImageBind MCM Motion-Verse
Closest papers by meaning
- 🎧MOSPA: Human Motion Generation Driven by Spatial Audio (NeurIPS 2025 · similarity 0.88)
- Everything2Motion: Synchronizing Diverse Inputs via a Unified Framework for Human Motion Synthesis (AAAI 2024 · similarity 0.87)
- MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators (AAAI 2024 · similarity 0.87)
- AMD: Autoregressive Motion Diffusion (AAAI 2024 · similarity 0.87)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.86)
- HumanTOMATO: Text-aligned Whole-body Motion Generation (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.