Research map / Vision-language and multimodal / Human and embodied modeling
Everything2Motion: Synchronizing Diverse Inputs via a Unified Framework for Human Motion Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Everything2Motion, a single transformer-based autoencoder framework with modality adapters, a query-aware latent autoencoder, and a pretrained motion-prior decoder that handles both text-to-motion (KIT-ML) and music-to-dance (AIST++).
Problem
Text-to-motion and music-to-dance synthesis are addressed by separate modality-specific models, which complicates unified deployment in animation platforms.
Approach
A unified framework with per-modality input adapters mapping into a shared latent space, a Perceiver-IO-style query-aware encoder-decoder with learnable latent and output queries, and a pretrained VAE decoder (prior motion distillation) that produces smooth skeleton sequences.
Key results
- On KIT-ML text-to-motion, E2M reports FID 1.060 vs 1.954 for MotionDiffuse and 2.770 for Guo et al.
- On AIST++ music-to-dance, E2M achieves the best Beat Align score of 0.248 but a worse FID of 38.23 than FACT (22.11).
- Inference for 120 text-to-motion frames takes 0.0904s vs 0.622s for Guo et al. on a 3090 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use E2M instead of MotionDiffuse | Text2Motion · KIT-ML | FID: 1.954 -> 1.060 | Helped |
| Use E2M instead of MotionDiffuse | Text2Motion · KIT-ML | R precision (Top 1): 0.417 -> 0.385 | Hurt |
| Use E2M instead of MotionDiffuse | Text2Motion · KIT-ML | Diversity: 11.10 -> 11.15 | Helped |
| Use E2M instead of FACT | Music2Dance · AIST++ | Beat Align: 0.221 -> 0.248 | Helped |
| Use E2M instead of DanceNet | Music2Dance · AIST++ | FID: 25.49 -> 38.23 | Hurt |
| Use E2M instead of FACT | Music2Dance · AIST++ | Diversity: 6.18 -> 6.68 | Helped |
| Add the Prior Motion Distillation Decoder (ablation) | E2M · KIT-ML | FID: 4.258 -> 1.060 | Helped |
| Add the pre-trained CLIP encoder (ablation) | E2M · KIT-ML | FID: 4.624 -> 1.060 | Helped |
| Use 64 instead of 128 latent codes, keeping 128 dimensions (ablation) | E2M Query-aware Autoencoder · KIT-ML | FID: 1.160 -> 2.086 | Hurt |
| Use 64 instead of 128 latent dimensions, keeping 128 codes (ablation) | E2M Query-aware Autoencoder · KIT-ML | FID: 1.160 -> 2.468 | Hurt |
| Use 256 instead of 128 latent dimensions, keeping 128 codes (ablation) | E2M Query-aware Autoencoder · KIT-ML | FID: 1.160 -> 1.243 | Hurt |
| Use 256 latent codes and dimensions instead of 128 each (ablation) | E2M Query-aware Autoencoder · KIT-ML | FID: 1.160 -> 1.147 | No clear effect |
Surprising findings
- On AIST++, the proposed unified model has a substantially higher FID (38.23) than FACT (22.11) and Li et al. is also worse than FACT, so the method does not win on all metrics.
- Ablation shows enlarging latent codes or channels to 256 degrades FID in some configurations, i.e., latent shape is sensitive.
Limitations
- Only two tasks (text-to-motion and music-to-dance) are evaluated; other modalities such as video or image are left to future work.
- Music2Dance FID is worse than FACT and the method is not best on all metrics.
- Ablation results are reported on KIT-ML only, and the table notes are inconsistent with the main results.
Experimental setup
Datasets: KIT-ML AIST++
Benchmarks: KIT-ML AIST++
Metrics: FID R precision Diversity MM-Dist Multimodality Beat Align inference time
Baselines: Test2Gesture Language2Pose MoCoGAN Guo et al. MotionDiffuse Li et al. DanceNet DanceRevolution FACT
Builds on: Perceiver-IO FACT CLIP MotionDiffuse TEMOS
Closest papers by meaning
- MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controls (AAAI 2025 · similarity 0.87)
- Seamless Human Motion Composition with Blended Positional Encodings (CVPR 2024 · similarity 0.87)
- 🎧MOSPA: Human Motion Generation Driven by Spatial Audio (NeurIPS 2025 · similarity 0.86)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.85)
- AMD: Autoregressive Motion Diffusion (AAAI 2024 · similarity 0.85)
- Towards Variable and Coordinated Holistic Co-Speech Motion Generation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.