Research map / Video / Video generation
GVMGen: A General Video-to-Music Generation Model with Hierarchical Attentions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GVMGen, an encoder-decoder video-to-music model using hierarchical spatial self-attention, spatial cross-attention with music queries, and temporal cross-attention feeding a pretrained MusicGen/Encodec decoder, plus a new evaluation model and a large mixed video-music dataset.
Problem
Generating background music that corresponds closely to video content and is diverse across video types and musical styles, given weak cross-modal alignment methods and limited, MIDI-centric datasets with low diversity.
Approach
Replaces rule-based variable mapping and LLM-based language bridges with hierarchical attention: CLIP ViT-L/14 spatial self-attention for visual features, learned music queries cross-attending to visual patches (averaged over queries), and temporal cross-attention from music tokens to visual features, decoded by a pretrained MusicGen decoder.
Key results
- Objective evaluation: GVMGen reaches FAD_vgg 7.45 and KLD 1.40 versus 11.23 and 1.98 for the best baseline M$^2$UGen, with CMR 0.88.
- Subjective evaluation with 20 non-experts and 10 experts: the full large model scores MVC 4.62 (non-expert) vs 2.85 for CMT and MR 5.98 vs 5.10 for M$^2$UGen.
- The evaluation model reports 99.4% accuracy on its test set, with a 3.75% error rate against 20 expert scores on 30 pairs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GVMGen instead of the strongest baseline on the evaluation set | GVMGen · Evaluation set | FAD_vgg: 11.23±2.63 -> 7.45±1.62 | Helped |
| Use GVMGen instead of the strongest baseline on the evaluation set | GVMGen · Evaluation set | KLD: 1.98±0.68 -> 1.40±0.55 | Helped |
| Use GVMGen instead of the strongest baseline on the evaluation set | GVMGen · Evaluation set | CMR: 0.88±0.07 -> 0.88±0.06 | No clear effect |
| Use GVMGen instead of the strongest baseline on the evaluation set | GVMGen · Evaluation set | TA: 0.83±0.07 -> 0.79±0.09 | Hurt |
| Replace ViT with ViViT for visual feature extraction (ablation) | GVMGen · Ablation test set | KLD: 1.07 -> 1.25 | Hurt |
| Replace ViT with ViViT for visual feature extraction (ablation) | GVMGen · Ablation test set | TA: 0.80 -> 0.77 | Hurt |
| Remove temporal cross-attention (ablation) | GVMGen · Ablation test set | FAD_vgg: 2.63 -> 16.71 | Hurt |
| Remove temporal cross-attention (ablation) | GVMGen · Ablation test set | CMR: 0.98 -> 0.35 | Hurt |
| Use 16 queries with average pooling instead of 4, 8, or 32 queries with average pooling (ablation) | GVMGen · Ablation test set | KLD: 1.20, 1.22, 1.26 -> 1.07 | Helped |
| Use average pooling rather than sum pooling for 16 queries (ablation) | GVMGen · Ablation test set | CMR: 0.94 -> 0.98 | Helped |
| Add the feature transformation module (ablation) | GVMGen (small model, CTM dataset) · Subjective evaluation, non-experts | OMQ: 3.29±0.12 -> 4.41±0.11 | Helped |
| Increase temporal cross-attention model size from small to large (ablation) | GVMGen (FTM, CTM dataset) · Subjective evaluation, non-experts | OMQ: 4.41±0.11 -> 4.57±0.11 | Helped |
Surprising findings
- Replacing ViT with ViViT for visual features reduced both generative similarity and music-video correspondence, which the authors attribute to temporal information loss.
- Without temporal cross-attention, FAD_vgg rises from about 2.6 to 16.71 and CMR drops to 0.35.
Limitations
- The dataset is not stated to be released, and the evaluation relies on a small number of samples (40 objective pieces, 32 subjective samples).
- Subjective evaluation uses relatively few expert listeners and samples; the evaluation model's agreement with humans is measured only on 30 pairs.
Experimental setup
Datasets: SymMV MuVi-Sync AIST++ TikTok Custom video-music dataset
Benchmarks: SymMV MuVi-Sync Custom test set (Other: movies and comics)
Metrics: FAD_vgg KLD CMR TA OMQ MVC MR
Baselines: CMT V2M M$^2$UGen NExT-GPT CoDi
Builds on: MusicGen CLIP Encodec Music Transformer M$^2$UGen
Closest papers by meaning
- VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling (CVPR 2025 · similarity 0.91)
- V2Meow: Meowing to the Visual Beat via Video-to-Music Generation (AAAI 2024 · similarity 0.90)
- Customized Condition Controllable Generation for Video Soundtrack (CVPR 2025 · similarity 0.87)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.86)
- HarmonySet: A Comprehensive Dataset for Understanding Video-Music Semantic Alignment and Temporal Synchronization (CVPR 2025 · similarity 0.86)
- MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.