Research map / Video / Video-language understanding
HarmonySet: A Comprehensive Dataset for Understanding Video-Music Semantic Alignment and Temporal Synchronization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HarmonySet, a 48K-pair video-music dataset with human-machine annotations on rhythm, emotion, theme, and culture, plus an open-ended and multiple-choice benchmark; fine-tuning VideoLLaMA2 on it improves video-music alignment reasoning.
Problem
Existing video-music datasets lack detailed annotations of semantic alignment and temporal synchronization, so multimodal LLMs give only surface-level interpretations of how music fits a video.
Approach
A multi-step human-machine annotation pipeline: humans mark synchronization timestamps and label four alignment dimensions, then Gemini 1.5 Pro expands these into detailed temporally grounded explanations; the resulting data supports instruction tuning and an open-ended plus multiple-choice benchmark.
Key results
- VideoLLaMA2 fine-tuned on HarmonySet improves Rhythm & Synchronization from 4.15 to 5.55 and Culture from 3.05 to 4.62 on HarmonySet-OE.
- Gemini-1.5 Pro scores 5.43 overall on R&S, the best untrained model, but humans reach 85-94% on HarmonySet-MC versus at most 54% for models.
- Training on human-verified annotations beats fully automated Gemini annotations at 10k samples (R&S 4.86 vs 4.59 overall).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune VideoLLaMA2 on HarmonySet | VideoLLaMA2 (Qwen2-7B) · HarmonySet-OE (Overall) | R & S: 4.15 -> 5.55 | Helped |
| Fine-tune VideoLLaMA2 on HarmonySet | VideoLLaMA2 (Qwen2-7B) · HarmonySet-OE (Overall) | C: 3.05 -> 4.62 | Helped |
| Train with HarmonySet annotations instead of fully automated annotations (10k) | VideoLLaMA2 (Qwen2-7B) · HarmonySet-OE (Overall) | R & S: 4.59 -> 4.86 | Helped |
| Train with HarmonySet annotations instead of fully automated annotations (10k) | VideoLLaMA2 (Qwen2-7B) · HarmonySet-OE (Overall) | T: 4.16 -> 4.70 | Helped |
| Train with HarmonySet annotations instead of fully automated annotations (10k) | VideoLLaMA2 (Qwen2-7B) · HarmonySet-OE (Overall) | E: 4.28 -> 4.66 | Helped |
| Train with HarmonySet annotations instead of fully automated annotations (10k) | VideoLLaMA2 (Qwen2-7B) · HarmonySet-OE (Overall) | C: 3.44 -> 3.89 | Helped |
| Increase training frames from 16 to 32 (ablation) | VideoLLaMA2 (HarmonySet) · HarmonySet-OE (Overall) | R & S: 5.55 -> 5.59 | Helped |
| Increase training frames from 16 to 64 (ablation) | VideoLLaMA2 (HarmonySet) · HarmonySet-OE (Overall) | R & S: 5.55 -> 5.49 | Hurt |
| Increase training frames from 16 to 64 (ablation) | VideoLLaMA2 (HarmonySet) · HarmonySet-OE (Overall) | T: 5.06 -> 4.94 | Hurt |
| Increase training frames from 16 to 64 (ablation) | VideoLLaMA2 (HarmonySet) · HarmonySet-OE (Overall) | C: 4.62 -> 4.53 | Hurt |
Surprising findings
- Using 64 frames gave worse results than 16 or 32 frames for VideoLLaMA2 fine-tuned on HarmonySet, attributed to short videos.
- VideoLLaMA2 fine-tuned on HarmonySet scored lower than the untrained model on rhythm-synchronization multiple-choice accuracy (10.63% vs 21.76%).
Limitations
- Videos are short (under one minute), limiting evaluation of longer temporal relationships.
- Only 58% of samples contain key timestamp annotations.
- Consensus evaluation relies on a 10% sample with subjective agreement labels.
Experimental setup
Datasets: HarmonySet
Benchmarks: HarmonySet-OE HarmonySet-MC
Metrics: LLM-judged score (1-10) for rhythm and synchronization, them multiple-choice accuracy
Baselines: Gemini-1.5 Pro VideoLLaMA2 video-SALMONN
Builds on: VideoLLaMA2 Gemini 1.5 Pro
Closest papers by meaning
- VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling (CVPR 2025 · similarity 0.86)
- GVMGen: A General Video-to-Music Generation Model with Hierarchical Attentions (AAAI 2025 · similarity 0.86)
- V2Meow: Meowing to the Visual Beat via Video-to-Music Generation (AAAI 2024 · similarity 0.85)
- LLark: A Multimodal Instruction-Following Language Model for Music (ICML 2024 · similarity 0.85)
- MoMu-Diffusion: On Learning Long-Term Motion-Music Synchronization and Correspondence (NeurIPS 2024 · similarity 0.85)
- CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction (ICML 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.