Research map / Vision-language and multimodal / Cross-modal learning
Music Gesture for Visual Sound Separation
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use explicit body and hand keypoints with semantic context | visual sound separation on MUSIC · MUSIC 2-Mix | SDR: 8.31 -> 10.12 | Helped |
| use explicit body and hand keypoints with semantic context | visual sound separation on MUSIC · MUSIC 3-Mix | SDR: 4.87 -> 5.41 | Helped |
| use explicit keypoint-based body dynamics | same-instrument trumpet duets · trumpet duet | SDR: 1.8 -> 4.9 | Helped |
| use explicit keypoint-based body dynamics | same-instrument flute duets · flute duet | SDR: 1.5 -> 5.3 | Helped |
| use explicit keypoint-based body dynamics | same-instrument piano duets · piano duet | SDR: 0.8 -> 3.8 | Helped |
| add the audio-visual fusion module (ablation) | Music Gesture · MUSIC 2-Mix | SDR: 9.64 -> 10.12 | Helped |
| add RGB context features (ablation) | Music Gesture · MUSIC 2-Mix | SDR: 10.22 -> 10.12 | Hurt |
| replace the proposed audio-visual fusion module with FiLM (ablation) | Music Gesture · MUSIC 2-Mix | SDR: +0.5dB SDR for the proposed module | Hurt |
| use keypoint-based body dynamics rather than trajectory and optical-flow-like cues | same-instrument violin and cello duets · violin and cello duets | human preference: 26% -> 74%; 28% -> 72% | Helped |
Closest papers by meaning
- iQuery: Instruments As Queries for Audio-Visual Sound Separation (CVPR 2023 · similarity 0.87)
- MoMu-Diffusion: On Learning Long-Term Motion-Music Synchronization and Correspondence (NeurIPS 2024 · similarity 0.85)
- HarmonySet: A Comprehensive Dataset for Understanding Video-Music Semantic Alignment and Temporal Synchronization (CVPR 2025 · similarity 0.84)
- MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models (CVPR 2024 · similarity 0.84)
- Continual Audio-Visual Sound Separation (NeurIPS 2024 · similarity 0.84)
- T-VSL: Text-Guided Visual Sound Source Localization in Mixtures (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.