Research map / Label-efficient and robust learning / Recognition and long-tail
Improving Sound Source Localization with Joint Slot Attention on Image and Audio
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes joint slot attention over image and audio features, with target/off-target slots, contrastive learning on target slots, and cross-modal attention matching, for self-supervised sound source localization.
Problem
Self-supervised sound source localization relies on single global audio and single local image embeddings, which are sensitive to background noise and off-target sounds and tend to localize only small discriminative parts of the source.
Approach
Joint slot attention with two shared learnable slots competitively decomposes image and audio features into target and off-target slots; only target slots are contrasted, and cross-modal attention matching aligns audio target queries with individual image features via stop-gradient intra-modal attention.
Key results
- On VGG-SS with VGGSound-144k training, the method reports cIoU 40.62 and AUC 41.66 without OGL, and 42.56 / 42.46 with OGL, best in that setting.
- On AVSBench (zero-shot, trained on VGGSound-144k), mIoU 31.33 vs 29.6 for Alignment and F-score 44.30 vs 35.9.
- Cross-modal retrieval on VGG-SS: Recall@1 of 30.61 audio-to-image and 31.70 image-to-audio, versus 16.47 and 20.09 for Alignment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add cross-modal attention matching (Lmatch) (ablation) | Ours · VGG-SS, VGGSound-144k | cIoU: 14.95 -> 36.23 | Helped |
| Add slot divergence loss (Ldiv) (ablation) | Ours · VGG-SS, VGGSound-144k | cIoU: 14.95 -> 18.69 | Helped |
| Add false-negative mitigation (ablation) | Ours · VGG-SS, VGGSound-144k | cIoU: 14.95 -> 15.03 | No clear effect |
| Add Ldiv and false-negative mitigation while retaining Lmatch (ablation) | Ours · VGG-SS, VGGSound-144k | cIoU: 36.23 -> 40.71 | Helped |
| Add false-negative mitigation while retaining Lmatch and Ldiv (ablation) | Ours · VGG-SS, VGGSound-144k | cIoU: 40.56 -> 40.71 | Helped |
| Add Ldiv while retaining Lmatch and false-negative mitigation (ablation) | Ours · VGG-SS, VGGSound-144k | cIoU: 36.14 -> 40.71 | Helped |
| Add Lmatch while retaining Ldiv and false-negative mitigation (ablation) | Ours · VGG-SS, VGGSound-144k | cIoU: 20.37 -> 40.71 | Helped |
| Share the initial slots across image and audio modalities (ablation) | Ours · VGG-SS, Flickr-144k training | cIoU: 34.80 -> 36.64 | Helped |
| Share the initial slots across image and audio modalities (ablation) | Ours · VGG-SS, VGGSound-144k training | cIoU: 33.11 -> 40.71 | Helped |
| Use the proposed method instead of Alignment | Sound source localization · AVSBench S4 | mIoU: 29.6 -> 31.33 | Helped |
| Use the proposed method instead of Alignment | Cross-modal retrieval · VGG-SS, VGGSound-144k | Recall@1, Audio → Image: 16.47 -> 30.61 | Helped |
| Use the proposed method instead of Alignment | Cross-modal retrieval · VGG-SS, VGGSound-144k | Recall@1, Image → Audio: 20.09 -> 31.70 | Helped |
Surprising findings
- Sharing initial slots across modalities improves results (e.g., VGG-SS cIoU 33.11 to 40.71 on VGGSound-144k in ablation).
Limitations
- Authors note the method targets single-source localization; multi-source SSL is left to future work.
- Gains on some settings are small or absent; the method does not win all settings (e.g., Flickr 144k with OGL lower than Alignment on some metrics).
Experimental setup
Datasets: Flickr-SoundNet VGG-Sound
Benchmarks: Flickr-SoundNet-Test VGG-SS AVSBench Heard 110 Unheard 110
Metrics: cIoU AUC mIoU F-score Recall@K
Baselines: Attention10k LVS EZ-VSL FNAC Alignment SLAVC HardPos SSL-TIE DMC NoPrior
Builds on: slot attention InfoNCE EZ-VSL Alignment (Senocak et al.) FNAC
Closest papers by meaning
- AV-SSAN: Audio-Visual Selective DOA Estimation Through Explicit Multi-Band Semantic-Spatial Alignment (AAAI 2026 · similarity 0.89)
- Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learning (CVPR 2023 · similarity 0.89)
- Object-aware Sound Source Localization via Audio-Visual Scene Understanding (CVPR 2025 · similarity 0.89)
- T-VSL: Text-Guided Visual Sound Source Localization in Mixtures (CVPR 2024 · similarity 0.89)
- Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledge (CVPR 2024 · similarity 0.86)
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.