Research map / Detection and segmentation / Open-set and few-shot detection
Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an iterative object identification (IOI) module with an object similarity-aware clustering (OSC) loss for multi-sound source localization that works without knowing the number of sound sources.
Problem
Existing multi-sound source localization methods require prior knowledge of the number of sound sources in a mixture, which limits their use in real-world settings.
Approach
An iterative procedure repeatedly picks the highest sound-associated visual cell, separates foreground from background via inner-product comparison with a negative cell, and excludes found regions; an OSC loss then pulls same-object cells together and pushes different objects and background apart, so the number of objects is inferred automatically.
Key results
- On VGGSound-Duet, the method reports CAP 32.5, PIAP 44.4, CIoU@0.3 46.9, AUC 29.2, versus AVGN's 21.9, 28.1, 26.2, 23.8.
- On MUSIC-Solo, AP 77.4, IoU@0.5 62.1, AUC 59.4, with the strongest single-source AUC among compared methods.
- Source counting accuracy of 90.79%, 83.21%, and 68.94% on VGGSound-Single, Duet, and Trio; inference time reduced by 28.3% versus AVGN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed method instead of AVGN | multi-source localization · MUSIC-Duet | PIAP: 57.2 -> 72.5 | Helped |
| use the proposed method instead of AVGN | multi-source localization · VGGSound-Duet | CIoU@0.3: 26.2 -> 46.9 | Helped |
| use the proposed method instead of AVGN | single-source localization · MUSIC-Solo | AUC: 48.5 -> 59.4 | Helped |
| use the proposed method instead of AVGN | single-source localization · VGGSound-Single | AUC: 42.3 -> 41.2 | Hurt |
| add Losc (ablation) | proposed method · VGGSound-Duet | CIoU@0.3: 25.3 -> 46.9 | Helped |
| add Losc (ablation) | proposed method · VGGSound-Duet | PIAP: 44.1 -> 44.4 | Helped |
| add Lavc (ablation) | proposed method · VGGSound-Duet | CIoU@0.3: 43.0 -> 46.9 | Helped |
| add Lavc (ablation) | proposed method · VGGSound-Duet | PIAP: 44.3 -> 44.4 | No clear effect |
| vary IOI thresholds τ1 and τ2 from 0.7 and 0.6 (ablation) | proposed method · VGGSound-Duet | CIoU@0.3: 46.9; 43.9; 45.0; 46.3; 46.8 | Mixed |
Surprising findings
- Table 2 has the proposed method's VGGSound-Single AUC (41.2) below several baselines' AUC for AVGN-era results, while its AP and IoU are best; the paper notes AUC is second-best.
- Using the audio-visual contrastive loss alone gives PIAP 44.1 but CIoU only 25.3, showing the OSC loss is needed for localization quality.
Limitations
- Sound source counting accuracy drops to 68.94% on the three-source VGGSound-Trio set.
- Relies on hand-set thresholds (tau1, tau2) and loss weights for clustering.
Experimental setup
Datasets: MUSIC VGGSound
Benchmarks: MUSIC-Solo MUSIC-Duet VGGSound-Single VGGSound-Duet VGGSound-Trio VGGSound-Mixed
Metrics: AP IoU@0.5 AUC CAP PIAP CIoU@0.3
Baselines: Attention10k OTS DMC CoarseToFine DSOL LVS EZ-VSL Mix-and-Localize AVGN
Builds on: LVS (Chen et al., CVPR 2021) AVGN Mix-and-Localize
Closest papers by meaning
- Object-aware Sound Source Localization via Audio-Visual Scene Understanding (CVPR 2025 · similarity 0.91)
- Mix and Localize: Localizing Sound Sources in Mixtures (CVPR 2022 · similarity 0.91)
- T-VSL: Text-Guided Visual Sound Source Localization in Mixtures (CVPR 2024 · similarity 0.89)
- Improving Sound Source Localization with Joint Slot Attention on Image and Audio (CVPR 2025 · similarity 0.86)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.85)
- Cyclic Learning for Binaural Audio Generation and Localization (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.