Research map / Video / Video-language understanding
MuseChat: A Conversational Music Recommendation System for Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 98K-quartet conversational music-for-video recommendation dataset and a MuseChat system (tri-modal contrastive fusion retriever plus a Vicuna-7B-based explanation generator) that refines music suggestions from user text prompts.
Problem
Video-based music recommendation ignores user preferences and cannot be refined through interaction or explained; existing systems are one-shot content-matching black boxes.
Approach
Treat the second-turn recommendation as metric learning in which fused video, candidate music, and user text prompt should land near the target music embedding, combined with an LLM sentence generator that conditions on music embeddings plus titles to justify picks.
Key results
- MuseChat reaches R@1 32.79 and SR 40.49 on 500-track pools, versus 25.71 R@1 for the best fusion baseline (Cross-Attn Fusion) and 22.61 for ViML.
- Human MOS overall score of 4.46 for MuseChat vs 3.07 for Vicuna-7B and 2.93 for Vicuna w/ Music.
- Removing the video branch drops R@1 to 8.12, the lowest among ablations, showing visual input is essential.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MuseChat's tri-modal recommendation and fusion approach instead of ViML | MuseChat · Music retrieval test pools | R@1: 22.61 -> 32.79 | Helped |
| use MuseChat's tri-modal recommendation and fusion approach instead of Cross-Attn Fusion | MuseChat · Music retrieval test pools | R@1: 25.71 -> 32.79 | Helped |
| add cross-attention fusion between music and text (ablation) | MuseChat music retrieval · Music retrieval test pools | R@1: 21.40 -> 25.71 | Helped |
| add self-attention layers before cross-attention fusion (ablation) | MuseChat music retrieval · Music retrieval test pools | R@1: 25.71 -> 32.79 | Helped |
| replace direct summation with cross-attention-based tri-modal fusion (ablation) | MuseChat music retrieval · Music retrieval test pools | R@1: 10.58 -> 32.79 | Helped |
| add the video modality (ablation) | MuseChat music retrieval · Music retrieval test pools | R@1: 8.12 -> 32.79 | Helped |
| add candidate music to video and text inputs (ablation) | MuseChat music retrieval · Music retrieval test pools | R@1: 22.67 -> 32.79 | Helped |
| add user prompt to video-only retrieval (ablation) | MVPt · Music retrieval test pools | R@1: 20.71 -> 22.67 | Helped |
| use both music title and music embeddings instead of music title alone (ablation) | MuseChat sentence generator · Music reasoning simulated conversations | BertScore (f1): 0.9453 -> 0.9676 | Helped |
| use both music title and music embeddings instead of music embeddings alone (ablation) | MuseChat sentence generator · Music reasoning simulated conversations | BertScore (f1): 0.9526 -> 0.9676 | Helped |
| use MuseChat's title-plus-embedding sentence generator instead of Vicuna-7B | MuseChat sentence generator · Human evaluation of music reasoning | Overall: 3.07 -> 4.46 | Helped |
| use MuseChat's title-plus-embedding sentence generator instead of Vicuna w/ Music | MuseChat sentence generator · Human evaluation of music reasoning | Overall: 2.93 -> 4.46 | Helped |
Surprising findings
- Vicuna w/ Music, using only audio embeddings, fails to identify correct music and artist names, motivating the use of titles as extra input.
Limitations
- Dialogue is limited to two turns in the dataset and experiments.
- Proprietary 3M-pair MVP training data is not public, and the candidate generator is a separate model.
- Dialogues are synthetic from GPT-3.5, and metadata is missing for many tracks.
Experimental setup
Datasets: YouTube8M MagnaTagATune Million Song Dataset
Benchmarks: Conversational Music Recommendation for Videos (held-out test pools)
Metrics: Recall@1 Recall@5 Recall@10 Median Rank Success Rate BERTScore AB Divergence L2 Distance Fisher-Rao Distance human MOS (correctness, musicality, clarity)
Baselines: VM-NET MVPt ViML Sum-Fusion Self-Attn Fusion Cross-Attn Fusion Vicuna-7B Vicuna w/ Music
Builds on: InfoNCE / contrastive multiview coding Vicuna LoRA CLIP Audio Spectrogram Transformer
Closest papers by meaning
- MusicRL: Aligning Music Generation to Human Preferences (ICML 2024 · similarity 0.84)
- Algorithmic Collective Action in Recommender Systems: Promoting Songs by Reordering Playlists (NeurIPS 2024 · similarity 0.83)
- HarmonySet: A Comprehensive Dataset for Understanding Video-Music Semantic Alignment and Temporal Synchronization (CVPR 2025 · similarity 0.82)
- MusER: Musical Element-Based Regularization for Generating Symbolic Music with Emotion (AAAI 2024 · similarity 0.81)
- JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal Parameters Tuning (AAAI 2025 · similarity 0.81)
- V2Meow: Meowing to the Visual Beat via Video-to-Music Generation (AAAI 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.