Research map / Vision-language and multimodal / Cross-modal learning
Object-aware Sound Source Localization via Audio-Visual Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an audio-visual sound source localization framework that uses an MLLM (InternVL 2.0-8B, training only) to generate foreground/background captions and trains with object-aware contrastive alignment and object region isolation losses.
Problem
Audio-visual sound source localization struggles in complex scenes where visually similar silent objects coexist with sound-making objects, since existing methods rely on simple audio-visual correspondence.
Approach
Use an MLLM at training time to produce detailed foreground (sound-making) and background (silent) captions, encode them as reference anchors, and apply an object-aware contrastive loss (OCA) plus an optimal-transport-based region isolation loss (ORI) to align audio-visual features; the MLLM is not used at inference.
Key results
- On MUSIC-Duet with ResNet18, CAP 61.4% vs 52.1% for NoPrior (CVPR'24); CIoU@0.3 45.9%, AUC 36.1%.
- On VGGSound-Duet with AudioCLIP backbone, CAP 47.1% vs 35.7% for T-VSL (CVPR'24).
- On MUSIC-Solo with ResNet18, IoU@0.5 71.1% vs 62.1% for NoPrior.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OCA loss (ablation) | Sound source localization model with ResNet18 · VGGSound-Duet test | CAP (%): 28.8 -> 43.8 | Helped |
| Add OCA loss (ablation) | Sound source localization model with ResNet18 · VGGSound-Duet test | CloU@0.3 (%): 31.4 -> 50.9 | Helped |
| Add ORI loss (ablation) | Sound source localization model with ResNet18 · VGGSound-Duet test | CAP (%): 28.8 -> 34.3 | Helped |
| Add ORI loss (ablation) | Sound source localization model with ResNet18 · VGGSound-Duet test | CloU@0.3 (%): 31.4 -> 49.1 | Helped |
| Add ORI loss alongside OCA (ablation) | Sound source localization model with ResNet18 · VGGSound-Duet test | CAP (%): 43.8 -> 45.9 | Helped |
| Add ORI loss alongside OCA (ablation) | Sound source localization model with ResNet18 · VGGSound-Duet test | CloU@0.3 (%): 50.9 -> 55.2 | Helped |
| Set OCA false-negative threshold τ to 0.7 rather than 0.1 (ablation) | Proposed Method · MUSIC-Duet and VGGSound-Duet | CAP (%): 54.1 -> 61.4; 36.1 -> 45.9 | Helped |
| Set OCA false-negative threshold τ to 0.7 rather than 0.4 (ablation) | Proposed Method · MUSIC-Duet and VGGSound-Duet | AUC (%): 32.4 -> 36.1; 33.5 -> 44.8 | Helped |
| Set OCA false-negative threshold τ to 0.7 rather than 1.0 (ablation) | Proposed Method · MUSIC-Duet and VGGSound-Duet | CAP (%): 59.2 -> 61.4; 39.9 -> 45.9 | Mixed |
Limitations
- Relies on MLLM-generated captions whose quality depends on the external model and prompt.
- Training requires an MLLM, adding cost, although it is removed at inference.
Experimental setup
Datasets: MUSIC VGGSound
Benchmarks: MUSIC-Solo MUSIC-Duet VGGSound-Single VGGSound-Duet VGGSound-Trio
Metrics: AP IoU@0.5 AUC CAP CIoU@0.3
Baselines: Attention10k OTS DMC CoarseToFIne DSOL LVS EZ-VSL Mix-and-Localize AVGN NoPrior T-VSL AudioCLIP
Builds on: LVS NoPrior T-VSL AudioCLIP Sinkhorn algorithm
Closest papers by meaning
- Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledge (CVPR 2024 · similarity 0.91)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.91)
- T-VSL: Text-Guided Visual Sound Source Localization in Mixtures (CVPR 2024 · similarity 0.91)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.90)
- Improving Sound Source Localization with Joint Slot Attention on Image and Audio (CVPR 2025 · similarity 0.89)
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.