Research map / Image generation / Autoregressive and latent generation
Enhancing Dance-to-Music Generation via Negative Conditioning Latent Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PN-Diffusion, a latent diffusion model for dance-to-music generation that trains a U-Net with positive conditioning (forward-played dance video) and negative conditioning (reverse-played video) via dual diffusion and reverse processes.
Problem
Generating music synchronized with rhythmic cues in dance videos (dance-to-music), where prior diffusion-based methods only use positive forward-played visual and motion conditioning.
Approach
Uses reverse-played dance videos as negative conditioning alongside forward-played positive conditioning, with positive and negative diffusion processes (noise +ε and −ε) and a bi-directional noise-prediction loss weighted by α to train a conditional latent diffusion U-Net.
Key results
- On AIST++, PN-Diffusion reaches BCS 97.64 and BHS 99.31, improving over best baseline LORIS (95.84 BCS, 95.09 BHS).
- On TikTok, PN-Diffusion achieves BCS 89.51 and BHS 91.73 versus best baseline CDCD (85.66 BCS, 85.83 BHS).
- User study on AIST++: MOS OVL 4.3 and REL 4.2 for PN-Diffusion versus 2.2/2.1 for CDCD and 2.0/2.0 for LORIS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use positive and negative conditioning together rather than the strongest baseline | PN-Diffusion · AIST++ | F1: +2.01 | Helped |
| Use positive and negative conditioning together rather than the strongest baseline | PN-Diffusion · TikTok | F1: +4.85 | Helped |
| Combine visual rhythm and motion conditioning rather than motion conditioning alone (ablation) | PN-Diffusion · AIST++ | F1: 91.59 -> 98.46 | Helped |
| Combine visual rhythm and motion conditioning rather than motion conditioning alone (ablation) | PN-Diffusion · TikTok | F1: 84.05 -> 90.60 | Helped |
| Combine visual rhythm and motion conditioning rather than visual rhythm conditioning alone (ablation) | PN-Diffusion · AIST++ | F1: 97.31 -> 98.46 | Helped |
| Combine visual rhythm and motion conditioning rather than visual rhythm conditioning alone (ablation) | PN-Diffusion · TikTok | F1: 86.30 -> 90.60 | Helped |
| Add negative conditioning to positive-only diffusion (ablation) | PN-Diffusion · AIST++ | F1: 96.58 -> 98.46 | Helped |
| Add negative conditioning to positive-only diffusion (ablation) | PN-Diffusion · TikTok | F1: 77.35 -> 90.60 | Helped |
| Add positive conditioning to negative-only diffusion (ablation) | PN-Diffusion · AIST++ | F1: 83.07 -> 98.46 | Helped |
| Add positive conditioning to negative-only diffusion (ablation) | PN-Diffusion · TikTok | F1: 73.91 -> 90.60 | Helped |
| Select reverse-played videos for negative conditioning instead of random videos (ablation) | PN-Diffusion · AIST++ | F1: 86.22 -> 98.46 | Helped |
| Select reverse-played videos for negative conditioning instead of random videos (ablation) | PN-Diffusion · TikTok | F1: 85.68 -> 90.60 | Helped |
Surprising findings
- Simply negating positive features as negative conditioning (DN-Diffusion) or using random mismatched videos (RN-Diffusion) performs worse than reverse-played videos.
- Combining original music with reverse-played dance video gave unsatisfactory synchronization, so negative conditioning was extracted using the original music paired with reverse-played video.
Limitations
- Evaluated only on two small dance datasets (AIST++ and TikTok) with 5-second segments.
- Negative conditioning relies on reverse-played video, a heuristic whose optimality is only validated empirically against a few alternatives.
- Subjective evaluation involves only 10 volunteers on a small sample.
Experimental setup
Datasets: AIST++ TikTok Dance-Music dataset
Benchmarks: AIST++ TikTok
Metrics: BCS BHS CSD HSD F1 FAD_v FAD_p FAD_c MOS (OVL, REL) Turing test
Baselines: FoleyMusic D2M-GAN CMT CDCD LORIS
Builds on: Latent Diffusion Model Stable Diffusion CDCD LORIS ST-GCN
Closest papers by meaning
- Bidirectional Autoregessive Diffusion Model for Dance Generation (CVPR 2024 · similarity 0.88)
- Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitives (CVPR 2024 · similarity 0.85)
- Diff-BGM: A Diffusion Model for Video Background Music Generation (CVPR 2024 · similarity 0.83)
- MoMu-Diffusion: On Learning Long-Term Motion-Music Synchronization and Correspondence (NeurIPS 2024 · similarity 0.83)
- Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators (NeurIPS 2026 · similarity 0.83)
- MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.