Research map / Video / Video generation
Diff-BGM: A Diffusion Model for Video Background Music Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the BGM909 video-music dataset and Diff-BGM, a latent-diffusion model that conditions piano-roll generation on video semantic and dynamic features via timestep-based feature selection and segment-aware cross-attention for video background music generation.
Problem
Generating background music for freestyle videos is hard due to the lack of suitable music-video datasets, difficulty controlling generation with different video aspects, and temporal alignment between video and music.
Approach
Use a diffusion model (built on Polyffusion) whose condition switches from semantic (caption) features to dynamic video features across denoising timesteps, so semantics shape melody first and dynamics shape rhythm later, with a segment-aware masked cross-attention layer for short-term audio-visual alignment.
Key results
- On BGM909, Diff-BGM improves music retrieval P@20 to 44.10% versus CMT's 36.40%, with P@5 at 13.28% versus 10.56%.
- In a 46-person user study, Diff-BGM was preferred over CMT at 77.8% overall by experts and 83.9% by non-experts.
- On SymMV, Diff-BGM reaches P@20 of 47.6% and average rank 19.4, against V-MusProd's 44.8% and 25.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add video features to unconditional generation (ablation) | Polyffusion-based Diff-BGM · BGM909 test set | PCHE: 3.189 -> 2.835 | Helped |
| add video features to unconditional generation (ablation) | Polyffusion-based Diff-BGM · BGM909 test set | P@20: 31.93% -> 43.20% | Helped |
| add feature selection to video-conditioned generation (ablation) | Polyffusion-based Diff-BGM · BGM909 test set | PCHE: 2.835 -> 2.721 | Helped |
| add segment-aware cross-attention (ablation) | Diff-BGM · BGM909 test set | P@20: 38.47% -> 44.10% | Helped |
| add segment-aware cross-attention (ablation) | Diff-BGM · BGM909 test set | PCHE: 2.721 -> 2.840 | Hurt |
| use only language features instead of only video features (ablation) | Diff-BGM feature selector · BGM909 test set | GPS: 0.514 -> 0.641 | Helped |
| use video features followed by language features rather than language followed by video (ablation) | Diff-BGM feature selector · BGM909 test set | PCHE: 2.781 -> 2.840 | Hurt |
| use language features followed by video features rather than video followed by language (ablation) | Diff-BGM feature selector · BGM909 test set | PCHE: 2.840 -> 2.781 | Helped |
| replace CMT with Diff-BGM | video background music generation · BGM909 test set | P@20: +7.70 percentage points (36.40% -> 44.10%) | Helped |
| replace V-MusProd with Diff-BGM | video background music generation · SymMV test set (our split) | P@20: 44.8 -> 47.6 | Helped |
| replace CMT with Diff-BGM | video background music generation · BGM909 user study | overall preference rate: 77.8% | Helped |
Surprising findings
- Adding segment-aware short-term masking improved retrieval but reduced music quality (e.g., PCHE and GPS slightly worse than the feature-selector variant).
- Human-created music scored only 3.48 in expert ratings, close to the proposed model's 3.33, indicating a gap remains to human-level composition.
Limitations
- Retrieval-based metrics rely on a pretrained Musicnn feature extractor and a retrieval pool, which may not fully capture musical quality.
- Music generation is limited to piano versions of songs (POP909-based piano music).
- Evaluation on SymMV uses a re-split test set that differs from the official split, limiting direct comparison of music-quality metrics.
Experimental setup
Datasets: BGM909 POP909 SymMV
Benchmarks: BGM909 test set SymMV test set
Metrics: PCHE GPS SI SC Diversity P@5 P@10 P@20 AR user preference
Baselines: CMT Riffusion V-MusProd Unconditional diffusion (Polyffusion)
Builds on: Polyffusion Latent Diffusion DDPM CMT
Closest papers by meaning
- Multi-Source Diffusion Models for Simultaneous Music Generation and Separation (ICLR 2024 · similarity 0.84)
- Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators (NeurIPS 2026 · similarity 0.84)
- Enhancing Dance-to-Music Generation via Negative Conditioning Latent Diffusion Model (CVPR 2025 · similarity 0.83)
- VideoScene: Distilling Video Diffusion Model to Generate 3D Scenes in One Step (CVPR 2025 · similarity 0.83)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.83)
- MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.