Research map / Video / Video generation
Controllable 3D Dance Generation Using Diffusion-Based Transformer U-Net
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Transformer U-Net diffusion model for 3D dance generation and a ControlNet-style control network that steers generated motion with 2D keypoint sequences, evaluated on AIST++ and in-the-wild videos.
Problem
Diffusion-based dance generation lacks controllability over body movements, which limits practical use; existing controllable diffusion methods are complex or need multiple training sessions.
Approach
A Transformer-decoder U-Net predicts dance motion from music via guided diffusion; after freezing it, a trainable copy of its downsampling layers with zero-convolution connections takes 2D keypoints as control input, following ControlNet.
Key results
- Control Network achieves FIDk 15.70 and FIDg 18.87 on AIST++, better FIDk than Ground Truth-level baselines EDGE (32.61) and Transformer U-Net (22.87), but BAS 0.22 is lower than Bailando's 0.23.
- Transformer U-Net reaches FIDg 22.76 and BAS 0.25, outperforming EDGE on FID metrics (25.08 FIDg) and achieving the best BAS among compared methods.
- Control signal reduces davg to 0.44 for the controlled framework versus 1.40 for the uncontrolled Transformer U-Net, and 0.24 vs 1.20 when keypoints match vs differ from ground truth.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a 2D-keypoint control network to the Transformer U-Net | Transformer U-Net · AIST++ | FIDg: 22.76 -> 18.87 | Helped |
| Add a 2D-keypoint control network to the Transformer U-Net | Transformer U-Net · AIST++ | FIDk: 22.87 -> 15.70 | Helped |
| Add a 2D-keypoint control network to the Transformer U-Net | Transformer U-Net · AIST++ | BAS: 0.25 -> 0.22 | Hurt |
| Add a 2D-keypoint control network to the Transformer U-Net | Transformer U-Net · AIST++ | davg: 1.40 -> 0.44 | Helped |
| Use control signals similar to the ground-truth motion rather than different control signals (ablation) | Control Network · AIST++ | davg: 1.20 -> 0.24 | Helped |
| Use Transformer U-Net rather than EDGE* | Music-conditioned dance generation · AIST++ | FIDk: 32.61 -> 22.87 | Helped |
| Use Transformer U-Net rather than Bailando* | Music-conditioned dance generation · AIST++ | FIDk: 28.16 -> 22.87 | Helped |
Surprising findings
- BAS metric exceeds ground truth for the generated motions, which the authors attribute to limitations of the BAS metric when generations closely match real data.
Limitations
- Depends heavily on quality of 2D keypoints; poor keypoints yield unrealistic motion.
- Generated foot movements and walking appear unrealistic since the model lacks physical simulation.
- Cannot effectively generate displacement (global translation) information.
Experimental setup
Datasets: PhantomDance AIST++
Benchmarks: AIST++
Metrics: FIDg FIDk BAS davg
Baselines: FACT Bailando EDGE LODGE Transformer U-Net
Builds on: ControlNet EDGE guided diffusion Bailando
Closest papers by meaning
- DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance (CVPR 2024 · similarity 0.87)
- Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitives (CVPR 2024 · similarity 0.87)
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.86)
- Move-in-2D: 2D-Conditioned Human Motion Generation (CVPR 2025 · similarity 0.86)
- Make Pixels Dance: High-Dynamic Video Generation (CVPR 2024 · similarity 0.86)
- Bidirectional Autoregessive Diffusion Model for Dance Generation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.