Research map / 3D vision / 3D generation
MEGADance: Mixture-of-Experts Architecture for Genre-Aware 3D Dance Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MEGADance, a two-stage music-to-3D-dance model that quantizes dance with FSQ plus kinematic-dynamic losses and generates latent codes with a genre-routed Mixture-of-Experts of Mamba-Transformer hybrid experts.
Problem
Music-driven 3D dance generation underuses genre conditioning (treated as weak auxiliary bias), causing poor music-motion synchronization and genre discontinuity, and VQ-VAE-based quantization suffers from codebook collapse.
Approach
Decouple choreography into shared Universal Experts (genre-agnostic) and genre-routed Specialized Experts (hard routing by genre label) within an MoE layer, each built from a Mamba-Transformer hybrid, operating on FSQ-discretized latent codes with forward-kinematics joint, velocity and acceleration losses.
Key results
- On FineDance, MEGADance achieves FID_k 50.00, FID_g 13.02, FID_s 2.52 and BAS 0.226, outperforming Bailando++, FineNet and Lodge.
- Ablation: FSQ replacing VQ-VAE raises joint MSE from 0.0069 to 0.0220, and FSQ reports 100% codebook utilization versus 75% for VQ-VAE.
- Reported runtime of 0.19 seconds per second of generated dance, claimed suitable for real-time use.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Specialized Experts (ablation) | MEGADance · FineDance | FID_s: 7.95 vs. 2.52 | Helped |
| add Universal Expert (ablation) | MEGADance · FineDance | FID_k: 54.50 → 50.00 | Helped |
| add Universal Expert (ablation) | MEGADance · FineDance | FID_g: 15.52 → 13.02 | Helped |
| add Mamba (ablation) | MEGADance · FineDance | FID_k: 56.29 vs. 50.00 | Helped |
| add kinematic loss (ablation) | MEGADance quantization · FineDance reconstruction | SMPL MSE: 0.0200 → 0.0238 | Helped |
| add kinematic loss (ablation) | MEGADance quantization · FineDance reconstruction | Joint MAE: 0.0469 → 0.0507 | Helped |
| add dynamic loss (ablation) | MEGADance quantization · FineDance reconstruction | Joint MSE: 0.0069 → 0.0073 | Helped |
| use FSQ instead of VQ-VAE (ablation) | MEGADance quantization · FineDance reconstruction | Joint MSE: 0.0220 vs. 0.0069 | Helped |
| use FSQ instead of VQ-VAE (ablation) | MEGADance quantization · codebook utilization | codebook utilization: 75% → 100% | Helped |
| use MEGADance instead of Bailando++ | music-to-dance generation · FineDance | FID_k: 54.79 -> 50.00 | Helped |
| use MEGADance instead of Lodge | music-to-dance generation · FineDance | FID_s: 5.22 -> 2.52 | Helped |
| use MEGADance instead of FineNet | music-to-dance generation · FineDance | BAS: 0.219 -> 0.226 | Helped |
Surprising findings
- Removing Universal Experts barely affects FID_s and BAS, while removing Specialized Experts causes the largest FID_s degradation (7.95 vs 2.52).
Limitations
- Genre labels impose rigid constraints and limit user intent expression.
- 3D motion capture data contains noise artifacts and the datasets are small, leading to overfitting risk.
Experimental setup
Datasets: FineDance AIST++
Benchmarks: FineDance AIST++
Metrics: FID_k FID_g FID_s DIV_k DIV_g DIV_s BAS user study DQ/DS/DC ACC F1
Baselines: Bailando++ FineNet Lodge
Builds on: Bailando++ VQ-VAE Mamba Finite Scalar Quantization
Closest papers by meaning
- DanceAnyWay: Synthesizing Beat-Guided 3D Dances with Randomized Temporal Contrastive Learning (AAAI 2024 · similarity 0.90)
- DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance (CVPR 2024 · similarity 0.88)
- Bidirectional Autoregessive Diffusion Model for Dance Generation (CVPR 2024 · similarity 0.88)
- Breaking the Group Size Barrier: Parameter-Efficient Group Dance Generation with Chain-of-Dancers (NeurIPS 2026 · similarity 0.87)
- POPDG: Popular 3D Dance Generation with PopDanceSet (CVPR 2024 · similarity 0.87)
- Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitives (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.