Research map / 3D vision / 3D generation
DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the DCM dataset of paired dance, camera and music data from anime (MMD) communities and DanceCamera3D, a transformer-based diffusion model with a body attention loss and strong-weak condition separation for CFG to synthesize camera movement from music and dance.
Problem
Synthesizing 3D camera movement conditioned on music and dance is unexplored, mainly due to the lack of paired dance-camera-music data and the multifaceted, human-centric nature of dance cinematography.
Approach
A transformer-based diffusion model for camera movement generation that adds a body attention loss penalizing camera motion that places dancer joints outside the view, and separates music and dance conditions in classifier-free guidance with distinct guidance weights.
Key results
- DanceCamera3D reaches FIDs 0.280 and FIDk 3.749 on DCM, versus 0.960 (FACT) and 2.368 (DanceRevolution) for shot and kinetic quality.
- DanceCamera3D lowers Dancer Missing Rate to 0.0025 compared with 0.0062 (DanceRevolution) and 0.0070 (FACT).
- In a 21-participant user study, DanceCamera3D beats baselines with win rates of at least 65.71%, and reaches 37.62% against ground truth.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DanceCamera3D instead of FACT | DanceCamera3D · DCM test set | FIDk: 5.205 -> 3.749 | Helped |
| Use DanceCamera3D instead of FACT | DanceCamera3D · DCM test set | DMR: 0.0070 -> 0.0025 | Helped |
| Add body attention loss (ablation) | DanceCamera3D · DCM test set | LCD: 0.310 -> 0.147 | Helped |
| Add body attention loss (ablation) | DanceCamera3D · DCM test set | DMR: 0.0899 -> 0.0025 | Helped |
| Add body attention loss (ablation) | DanceCamera3D · DCM test set | FIDk: 4.022 -> 3.749 | Helped |
| Add body attention loss (ablation) | DanceCamera3D · DCM test set | Distk: 1.421 -> 1.631 | Helped |
| Add body attention loss (ablation) | DanceCamera3D · DCM test set | Dists: 1.671 -> 1.326 | Hurt |
| Increase guidance weights for conditions | DanceCamera3D CFG · DCM test set | camera movement diversity | Mixed |
| Increase guidance weights for conditions | DanceCamera3D CFG · DCM test set | camera movement quality | Mixed |
| Apply guidance separately to dance and music conditions | DanceCamera3D CFG · DCM test set | trade-off control | Helped |
Surprising findings
- The body attention loss removal increases shot diversity because more frames lack the dancer on screen, which distorts the shot feature distribution.
Limitations
- Ground-truth camera movements from animators have more granular control and fewer artifacts than generated results.
- Dataset is small (108 sequences, 193 minutes) and drawn from anime MMD community resources.
Experimental setup
Datasets: DCM
Benchmarks: DCM
Metrics: FIDk FIDs Distk Dists DMR LCD user study win rate
Baselines: DanceRevolution FACT DanceCamera3D w/o Lba
Builds on: DDPM classifier-free guidance FACT DanceRevolution
Closest papers by meaning
- DanceAnyWay: Synthesizing Beat-Guided 3D Dances with Randomized Temporal Contrastive Learning (AAAI 2024 · similarity 0.90)
- MEGADance: Mixture-of-Experts Architecture for Genre-Aware 3D Dance Generation (NeurIPS 2025 · similarity 0.88)
- Controllable 3D Dance Generation Using Diffusion-Based Transformer U-Net (AAAI 2025 · similarity 0.87)
- Bidirectional Autoregessive Diffusion Model for Dance Generation (CVPR 2024 · similarity 0.87)
- MotionMix: Weakly-Supervised Diffusion for Controllable Motion Generation (AAAI 2024 · similarity 0.86)
- DisCo: Disentangled Control for Realistic Human Dance Generation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.