Research map / Video / Video generation
V2Meow: Meowing to the Visual Beat via Video-to-Music Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes V2Meow, a multi-stage autoregressive (AudioLM-style) video-to-music generator trained on 5k hours of music videos using general-purpose visual features, with optional MuLan text style control.
Problem
Generating high-fidelity music waveforms conditioned on arbitrary in-the-wild video, rather than domain-specific inputs such as dance motion or instrument performances, where audio-visual alignment is weak and training data is limited.
Approach
Learn globally aligned video-music correspondence directly from paired music videos without explicit rhythm or semantic modeling, by mapping general pre-trained visual features (CLIP, I3D flow, ViT-VQGAN tokens) to w2v-BERT semantic tokens in a first stage and then to SoundStream acoustic tokens, with MuLan embeddings for optional text control.
Key results
- On MV100K, V2Meow CLIP+I3D achieves the highest MCC (0.343 semantic-only, 0.419 with acoustic modeling) and V2Meow VIT+I3D reaches 83.8% visual relevance vs CMT's 20.6% in human pairwise study.
- On MusicCaps, V2Meow variants achieve roughly 20-30% higher visual relevance than Riffusion and MUBERT, with lower FAD (e.g., 0.328 vs 0.760 for Riffusion).
- Zero-shot on AIST++ dance videos, V2Meow CLIP+I3D (10s) reaches beat F1 91.5, exceeding CDCD Step-Intra (85.5), D2M-GAN (86.4) and CMT (84.5) which were trained on or use motion inputs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add video-frame conditioning to text conditioning | MusicCaps text-to-music generation · MusicCaps | Visual Relevance | Helped |
| combine CLIP and I3D Flow visual features (ablation) | V2Meow · MV100K | MCC: 0.343/0.419 | Helped |
| combine CLIP and I3D Flow visual features (ablation) | V2Meow · MV100K | Visual Relevance: 79.2% | Mixed |
| combine ViT-VQGAN tokens and I3D Flow features (ablation) | V2Meow · MusicCaps | Visual Relevance: 71.8% | Helped |
| add video conditioning during acoustic modeling (ablation) | V2Meow · MV100K | MCC: 0.343/0.419 | Helped |
| predict acoustic tokens directly from CLIP visual features, without semantic tokens (ablation) | V2Meow-CLIP · MV100K | FAD VGG: 2.779 to 3.331 | Hurt |
| predict acoustic tokens directly from CLIP visual features, without semantic tokens (ablation) | V2Meow-CLIP · MusicCaps | MCC: 0.377 to 0.275 | Hurt |
| use CLIP+I3D rather than CLIP alone (ablation) | V2Meow · MV100K human evaluation | Visual Relevance: 47.2% to 63.0% | Helped |
| use CLIP+I3D rather than CLIP alone (ablation) | V2Meow · MV100K human evaluation | Music Preference: 49.0% to 61.8% | Helped |
Surprising findings
- Directly predicting acoustic tokens from CLIP features (skipping semantic tokens) degrades FAD VGGish from 2.779 to 3.331 and semantic alignment, showing the semantic stage is essential.
- Combining visual features does not always help: VIT-VQGAN+I3D scores lower than CLIP+I3D on several metrics, and CLIP alone beats CLIP+I3D on Punk Rock.
Limitations
- Visual features sampled at 1 fps, so fast motion below the sampling rate is not captured.
- Training data (MV100K) is genre-skewed and contains gender/ethnic stereotypes, which the paper flags as a bias risk.
- Evaluation relies on human studies with limited participants and on proxy metrics (FAD, KLD, MCC) that may not reflect musical quality.
Experimental setup
Datasets: MV100K AudioSet MusicCaps AIST++
Benchmarks: MV100K MusicCaps AIST++
Metrics: FAD TRILL FAD VGGish KL divergence MuLan cycle consistency visual relevance win rate music preference win rate beat coverage beat hit F1
Baselines: CMT Riffusion MUBERT D2M-GAN CDCD Step-Intra
Builds on: AudioLM MusicLM SoundStream w2v-BERT MuLan
Closest papers by meaning
- VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling (CVPR 2025 · similarity 0.92)
- GVMGen: A General Video-to-Music Generation Model with Hierarchical Attentions (AAAI 2025 · similarity 0.90)
- Customized Condition Controllable Generation for Video Soundtrack (CVPR 2025 · similarity 0.89)
- Audio-Sync Video Generation with Multi-Stream Temporal Control (NeurIPS 2025 · similarity 0.87)
- FilmComposer: LLM-Driven Music Production for Silent Film Clips (CVPR 2025 · similarity 0.87)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.