Research map / Vision-language and multimodal / Human and embodied modeling
UniMuMo: Unified Text, Music, and Motion Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniMuMo, a unified text-music-motion generation transformer built by fine-tuning MusicGen with motion tokens in Encodec's music codebook, trained on rhythm-aligned music-motion data.
Problem
Generating music, motion, and text in arbitrary combinations within one model, despite the lack of time-synchronized music-motion-text datasets and the modality gap between representations.
Approach
Aligns unpaired music and motion by rhythmic beats with DTW and warping, encodes motion with a frozen pretrained music RVQ codebook (Encodec), and trains a MusicGen-based decoder on parallel music-motion joint generation with separate motion embedder, FFN (MoE-style), and head, so that music-to-motion, motion-to-music, and text-conditioned tasks work zero-shot.
Key results
- Text-to-music on MusicCaps: UniMuMo FAD_VGG 5.93 and KL 1.99 with CLAP 0.27, behind MusicGen (300M) at 4.9/1.42 and AudioLDM 2 at 3.13/1.17.
- Motion-conditioned music on AIST++: Beats Hit 88.4 for UniMuMo versus 90.7 for CDCD (best), and Beats Coverage 93.0 versus 93.9.
- Mean L1 distance between music beats and nearest visual beats drops from 6.34 to 1.78 after alignment; user study gives 3.95 aligned versus 3.26 random.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Align motion to music using beat detection and dynamic time warping | randomly paired music and motion clips · 300 sampled 10-second music clips | Mean L1 beat distance: 6.34 to 1.78 | Helped |
| Use an independent motion VQ-VAE without initializing the motion embedder from MusicGen, instead of shared music codebooks (ablation) | UniMuMo · MusicCaps | FAD_VGG: 5.93 -> 6.75 | Hurt |
| Use an independent motion VQ-VAE and initialize the motion embedder from MusicGen, instead of shared music codebooks (ablation) | UniMuMo · MusicCaps | FAD_VGG: 5.93 -> 6.79 | Hurt |
| Remove the separate motion embedder and MoE structure (ablation) | UniMuMo · Music4All | Dist_k: 10.54 -> 6.03 | Hurt |
| Train with a mixture of music-motion generation, music-to-motion, and motion-to-music tasks instead of the single joint-generation task (ablation) | UniMuMo · Music4All | Beat Alignment: 0.24 -> 0.22 | Hurt |
| Train UniMuMo from scratch instead of fine-tuning a pre-trained model (ablation) | UniMuMo · MusicCaps | FAD_VGG: 5.93 -> 7.10 | Hurt |
Surprising findings
- Training on vocal-containing music yields better quantitative FAD/KL scores but perceptually poor vocals, while vocal-free fine-tuning degrades audio quality for MusicGen as well.
- Initializing from pretrained MusicGen is important: training from scratch needed about 30K iterations versus 15K with pretraining and gave worse scores.
Limitations
- Text-to-music quality lags SOTA, attributed to vocal-free training data quality from Demucs separation.
- Motion captioning on HumanML3D lags the best baseline since only 50% of motion data comes from HumanML3D and motions are beat-augmented.
- Text-conditioned dance has inferior dance quality since no ground truth music exists for reference.
Experimental setup
Datasets: Music4All AIST++ DanceDB HumanML3D MusicCaps MusicQA
Benchmarks: MusicCaps AIST++ HumanML3D MusicQA
Metrics: FAD_VGG KL CLAP Beats Coverage Beats Hit Dist_k Dist_g Beat Align. BLEU METEOR
Baselines: Riffusion Mousai MusicLM MusicGen AudioLDM 2 Dance2Music Foley Music CMT D2M-GAN CDCD Bailando FACT
Builds on: MusicGen Encodec T5 BLIP Bailando
Closest papers by meaning
- MoMu-Diffusion: On Learning Long-Term Motion-Music Synchronization and Correspondence (NeurIPS 2024 · similarity 0.87)
- A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization (NeurIPS 2026 · similarity 0.86)
- UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities (NeurIPS 2026 · similarity 0.86)
- MuPT: A Generative Symbolic Music Pretrained Transformer (ICLR 2025 · similarity 0.86)
- MIDI-GPT: A Controllable Generative Model for Computer-Assisted Multitrack Music Composition (AAAI 2025 · similarity 0.85)
- UniAudio: Towards Universal Audio Generation with Large Language Models (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.