Research map / Video / Video generation
VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 360K-pair video-to-music dataset (V2M) and proposes VidMuse, which predicts EnCodec music tokens autoregressively from long-term and short-term visual features.
Problem
Generating music conditioned solely on video across diverse genres, where prior work is limited to dance videos, symbolic MIDI, or small/narrow datasets.
Approach
A Long-Short-Term Visual module fuses clip-level (short-term) features with video-level (long-term) context via cross-attention, feeding a transformer decoder that predicts discrete audio codec tokens end-to-end.
Key results
- On V2M-bench, VidMuse reaches KL 0.734, FD 29.946, FAD 2.459 and ImageBind 0.202, beating M2UGen (KL 0.997, FAD 5.104) and VM-NET.
- Ablation: removing the long-term module (VidMuse-STM) raises KL from 0.738 to 0.898 and FAD from 2.369 to 4.915.
- On AIST++ VidMuse gets BHS 96.5 and F1 98.2 versus V2Meow's 84.4 and 91.5 on those metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VidMuse instead of VM-NET | Video-to-music generation · V2M-bench | KL: 0.899 -> 0.734 | Helped |
| use VidMuse instead of M2UGen* | Video-to-music generation · V2M-bench | FD: 52.041 -> 29.946 | Helped |
| use VidMuse instead of Caption2Music | Video-to-music generation · V2M-bench | ImageBind: 0.191 -> 0.202 | Helped |
| use VidMuse instead of V2Meow | Video-to-music generation · AIST++ | F1: 91.5 -> 98.2 | Helped |
| use VidMuse instead of LORIS | Video-to-music generation · LORIS | F1: 94.5 -> 95.9 | Helped |
| remove long-term modeling (ablation) | VidMuse · V2M-bench | KL: 0.738 -> 0.898 | Hurt |
| use only long-term modeling (ablation) | VidMuse · V2M-bench | KL: 0.738 -> 0.858 | Hurt |
| use CAQ with short-term features first, then long-term features (ablation) | VidMuse · V2M-bench | KL: 0.738 -> 0.843 | Hurt |
| use CAQ with long-term features first, then short-term features (ablation) | VidMuse · V2M-bench | KL: 0.738 -> 0.919 | Hurt |
| replace LSTV-Module with a SlowFast-like mechanism (ablation) | VidMuse · V2M-bench | KL: 0.738 -> 1.511 | Hurt |
| replace CLIP visual encoder with ViViT (ablation) | VidMuse · V2M-bench | KL: 0.753 -> 0.822 | Hurt |
| replace CLIP visual encoder with VideoMAE (ablation) | VidMuse · V2M-bench | KL: 0.753 -> 0.778 | Hurt |
Surprising findings
- Local (short-term) information contributes more to generation than global (long-term) information alone: the long-term-only variant performs worse on FAD (16.074) than the short-term-only variant (4.915).
Limitations
- Relies on the EnCodec codec, which introduces noticeable reconstruction loss for some genres, limiting audio quality.
- Only visual conditioning is explored; no other control inputs such as text or style.
Experimental setup
Datasets: V2M V2M-20K HIMV-200K YouTube-8M AIST++ TikTok LORIS benchmark
Benchmarks: V2M-bench AIST++ TikTok LORIS benchmark
Metrics: KL FD FAD density coverage ImageBind score BCS BHS F1 user preference A/B
Baselines: Caption2Music Video2Music CMT M2UGen VM-NET V2Meow LORIS D2M-GAN
Builds on: MusicGen EnCodec CLIP ImageBind
Closest papers by meaning
- V2Meow: Meowing to the Visual Beat via Video-to-Music Generation (AAAI 2024 · similarity 0.92)
- GVMGen: A General Video-to-Music Generation Model with Hierarchical Attentions (AAAI 2025 · similarity 0.91)
- Customized Condition Controllable Generation for Video Soundtrack (CVPR 2025 · similarity 0.87)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.87)
- Audio-Sync Video Generation with Multi-Stream Temporal Control (NeurIPS 2025 · similarity 0.87)
- FilmComposer: LLM-Driven Music Production for Silent Film Clips (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.