Research map / Vision-language and multimodal / Cross-modal learning
T-VSL: Text-Guided Visual Sound Source Localization in Mixtures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes T-VSL, which uses AudioCLIP's tri-modal text-audio-image embedding space and class text prompts as coarse guidance to disentangle per-source audio-visual correspondence for visual sound source localization in single- and multi-source mixtures.
Problem
Visual sound source localization in multi-source mixtures, where per-source audio-visual correspondence cannot be learned directly because individual source sounds are not available during training.
Approach
Uses the text modality of the AudioCLIP joint embedding space as coarse categorical guidance: detected class text embeddings condition audio and visual cross-attention blocks to separate per-source features, followed by an audio-visual correspondence block and contrastive alignment, trained with weak video-level labels.
Key results
- On single-source VGGSound-Single, T-VSL reaches 48.1 AP and 53.7 IoU@0.5, outperforming the prior SOTA Alignment (45.3 AP, 50.8 IoU).
- On multi-source VGGSound-Duet, T-VSL achieves 35.7 CAP and 40.1 CIoU@0.3, +2.3 CIoU over AVGN.
- Zero-shot transfer with a 50%-50% class split yields 55.8 CIoU on MUSIC-Duet versus 50.9 for Mix-and-Localize.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the AVC block to mixture patch tokens (ablation) | T-VSL · VGGSound-Single | AP: 42.8 -> 43.9 | Helped |
| Add the AVC block to mixture patch tokens (ablation) | T-VSL · VGGSound-Duet | CAP: 28.4 -> 29.7 | Helped |
| Add audio conditioning after audio-visual instance detection (ablation) | T-VSL · VGGSound-Single | AP: 43.9 -> 45.7 | Helped |
| Add audio conditioning after audio-visual instance detection (ablation) | T-VSL · VGGSound-Duet | CAP: 29.7 -> 32.8 | Helped |
| Add image conditioning after audio-visual instance detection (ablation) | T-VSL · VGGSound-Single | AP: 43.9 -> 46.2 | Helped |
| Add image conditioning after audio-visual instance detection (ablation) | T-VSL · VGGSound-Duet | CAP: 29.7 -> 33.3 | Helped |
| Use both audio and image conditioning with AVC (ablation) | T-VSL · VGGSound-Single | AP: +5.3 AP | Helped |
| Use both audio and image conditioning with AVC (ablation) | T-VSL · VGGSound-Duet | CAP: +7.3 CAP | Helped |
| Increase learnable prompt length from 0 to 8 (ablation) | T-VSL · VGGSound-Single | AP: 48.1 -> 50.3 | Helped |
| Increase learnable prompt length from 0 to 16 (ablation) | T-VSL · VGGSound-Duet | CIoU@0.3: 40.1 -> 42.9 | Helped |
| Use learnable text prompts (ablation) | T-VSL · unseen classes across datasets | zero-shot localization | Hurt |
| Use T-VSL instead of Alignment | single-source localization · VGGSound-Single | IoU@0.5: +2.9 | Helped |
Surprising findings
- Many single-source localization methods lose performance relative to the plain AudioCLIP baseline when applied to multi-source mixtures, and some also degrade on single-source data with modified contrastive losses.
Limitations
- Learnable text prompts improve performance but limit zero-shot use to unseen classes.
- Relies on class-level text prompts, which cannot distinguish multiple silent instances of the same class.
- Several datasets became smaller due to video unavailability, so results use reduced splits.
Experimental setup
Datasets: MUSIC VGGSound VGGSound-Instruments
Benchmarks: MUSIC-Solo MUSIC-Duet VGGSound-Single VGGSound-Duet VGGSound-Instruments
Metrics: AP IoU AUC CAP CIoU zero-shot IoU/CIoU
Baselines: OTS CoarsetoFine LVS EZ-VSL Mix-and-Localize DSOL MarginNCE FNAC AVGN Alignment AudioCLIP Wav2CLIP
Builds on: AudioCLIP CLIP Mix-and-Localize AVGN
Closest papers by meaning
- Object-aware Sound Source Localization via Audio-Visual Scene Understanding (CVPR 2025 · similarity 0.91)
- Improving Sound Source Localization with Joint Slot Attention on Image and Audio (CVPR 2025 · similarity 0.89)
- Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledge (CVPR 2024 · similarity 0.89)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.88)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.87)
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.