Research map / Video / Video generation
MoMu-Diffusion: On Learning Long-Term Motion-Music Synchronization and Correspondence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoMu-Diffusion, a motion-music framework with a contrastively aligned VAE (BiCoR-VAE) and a Transformer diffusion model with cross-guidance sampling for cross-modal, joint, and variable-length generation.
Problem
Long-term, beat-synchronized generation of music from motion and motion from music is computationally expensive on raw sequences and prior work treats the two directions separately, with weak temporal alignment over long clips.
Approach
Learn modality-aligned latent spaces for motion and mel-spectrogram music with a rhythmic contrastive loss built on a kinematic amplitude indicator (from motion directogram differences), then train latent diffusion Transformers in both directions and combine them at sampling time via cross-guidance for joint generation without extra training.
Key results
- On Floor Exercise-25s, MoMu-Diffusion reaches F1 71.4 vs 62.7 for LORIS (beat-matching).
- On AIST++ music-to-motion, FID 7.3 vs 17.3 for D2M and Mean KLD 4.9 vs 12.6 for DiffGesture.
- Joint generation with T_c=0.5T achieves FAD 8.1 and motion F1 45.4, comparable to cross-modal models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoMu-Diffusion instead of LORIS | Motion-to-music generation · Floor Exercise-25s | F1: 62.7 -> 71.4 | Helped |
| Use MoMu-Diffusion instead of LORIS | Motion-to-music generation · Floor Exercise-50s | F1: 58.9 -> 65.3 | Helped |
| Use MoMu-Diffusion instead of LORIS | Motion-to-music generation · Figure Skating-25s | F1: 62.7 -> 69.0 | Helped |
| Use MoMu-Diffusion instead of LORIS | Motion-to-music generation · Figure Skating-50s | F1: 58.9 -> 64.0 | Helped |
| Use MoMu-Diffusion instead of DiffGesture | Music-to-motion generation · AIST++ Dance | F1: 33.4 -> 46.2 | Helped |
| Use MoMu-Diffusion instead of DiffGesture | Music-to-motion generation · BHS Dance | F1: 50.3 -> 59.5 | Helped |
| Use directional-vector motion encoding instead of spatial-position encoding (ablation) | MoMu-Diffusion · AIST++ Dance | FAD: 8.1 -> 10.9 | Hurt |
| Replace mel-spectrogram encoding with raw-waveform encoding (ablation) | MoMu-Diffusion · AIST++ Dance | FAD: 8.1 -> 12.8 | Hurt |
| Remove rhythmic contrastive learning (RCL) (ablation) | MoMu-Diffusion · AIST++ Dance | F1: 98.1 -> 93.1 | Hurt |
| Replace the Feed-Forward Transformer with a U-Net backbone (ablation) | MoMu-Diffusion · AIST++ Dance | FAD: 8.9 -> 11.0 | Hurt |
| Set cross-guidance timestep Tc=0.9T instead of Tc=0.5T (ablation) | MoMu-Diffusion joint generation · AIST++ Dance | FAD: 8.1 -> 14.8 | Hurt |
| Set cross-guidance timestep Tc=0.1T instead of Tc=0.5T (ablation) | MoMu-Diffusion joint generation · AIST++ Dance | F1: 96.5 -> 85.5 | Hurt |
Surprising findings
- Applying cross-guidance too early (T_c=0.9T) degrades joint generation sharply (FAD 14.8, motion F1 25.8), since predicted latents are too noisy.
Limitations
- Scaling behavior not tested on very large motion-music datasets due to limited data and compute.
- Relies on OpenPose keypoint extraction and mel-spectrogram preprocessing, which may accumulate errors.
- Generated skeletons can have abnormal bone lengths (failure cases mitigated by post-processing).
Experimental setup
Datasets: AIST++ Floor Exercise (GymNet/Ginegym) Figure Skating (FisV, FS1000) BHS Dance LORIS benchmark
Benchmarks: AIST++ Dance Floor Exercise Figure Skating BHS Dance
Metrics: BCS CSD BHS HSD F1 FAD Diversity FID Mean KLD
Baselines: Foley Music CMT D2M-GAN CDCD LORIS D2M DiffGesture Diff-Foley MM-Diffusion
Builds on: Dancing2Music LORIS classifier-free guidance Diff-Foley DiT
Closest papers by meaning
- Bidirectional Autoregessive Diffusion Model for Dance Generation (CVPR 2024 · similarity 0.88)
- Fast Timing-Conditioned Latent Audio Diffusion (ICML 2024 · similarity 0.88)
- Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models (ICLR 2024 · similarity 0.87)
- UniMuMo: Unified Text, Music, and Motion Generation (AAAI 2025 · similarity 0.87)
- JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation (AAAI 2025 · similarity 0.87)
- DanceAnyWay: Synthesizing Beat-Guided 3D Dances with Randomized Temporal Contrastive Learning (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.