Research map / Vision-language and multimodal / Cross-modal learning
Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a self-supervised mix-and-separate model (correspondence + disentanglement contrastive losses on mixed audio) that simultaneously grounds speech and non-speech sounds in images, and introduces Extended-IS3 for evaluating this.
Problem
Existing audio-visual grounding models handle speech or non-speech sounds independently or sequentially, not overlapping mixtures, and suffer from information leakage between audio-type heads.
Approach
Uses a mix-and-separate contrastive framework with per-audio-type heads in shared audio and visual encoders, where a correspondence loss on clean audio and a disentanglement loss on audio mixtures jointly train embeddings for speech and sound without generating separated audio.
Key results
- On Extended-IS3 simultaneous grounding, the model reaches 43.4 mAP for sound and 58.6 mAP for speech segmentation, vs DenseAV's 30.7 and 53.4.
- With mixed audio, the sound-head total segmentation mAP is 39.3 vs DenseAV's 30.5 (speech head 41.6 vs 33.0).
- On AudioSet cross-modal retrieval (clean), the model reaches 45.5 I→A and 46.6 A→I R@10 vs DenseAV's 41.8 and 43.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the mix-and-separate method instead of DenseAV | audio-visual grounding · Extended IS3, simultaneous sound segmentation | mAP: 30.7 -> 43.4; +12.7 mAP | Helped |
| use the mix-and-separate method instead of DenseAV | audio-visual grounding · Extended IS3, simultaneous speech segmentation | mAP: 53.4 -> 58.6; +5.2 mAP | Helped |
| use the mix-and-separate method instead of DenseAV | audio-visual grounding · ADE20K Speech and Sound, sound SemSeg Total | mAP: 30.3 -> 36.7 | Helped |
| use the mix-and-separate method instead of DenseAV | audio-visual grounding · ADE20K Speech and Sound, speech SemSeg Total | mAP: 43.0 -> 42.1 | Hurt |
| use the mix-and-separate method instead of DenseAV | audio-visual grounding · ADE20K Speech and Sound, mixed sound SemSeg Sound Head | mAP: 18.7 -> 28.4 | Helped |
| use the mix-and-separate method instead of DenseAV | audio-visual grounding · ADE20K Speech and Sound, mixed speech SemSeg Speech Head | mAP: 33.0 -> 41.6 | Helped |
| add the correspondence loss (ablation) | Ours · AudioSet, Sound Head retrieval | R@10: 30.9 -> 51.2; 32.0 -> 50.0 | Helped |
| add the disentanglement loss (ablation) | Ours · Places, Sound Head retrieval | R@10: 23.7 -> 2.3; 42.1 -> 10.5 | Helped |
Surprising findings
- The model's speech-prompted segmentation is slightly lower than DenseAV on clean speech, and it is outperformed by ImageBind on AudioSet retrieval.
Limitations
- Speech-prompted grounding is slightly below DenseAV on clean speech benchmarks.
- Extended-IS3 speech samples are synthetic (gTTS) rather than natural human speech.
- Two audio types only (K=2 heads), limiting generality to more sound categories.
Experimental setup
Datasets: AudioSet PlacesAudio ADE20K Speech and Sound (DenseAV test set) IS3 Extended-IS3
Benchmarks: Extended-IS3 ADE20K Speech and Sound Places/PlacesAudio retrieval AudioSet retrieval
Metrics: mAP mIoU R@10 Pred.Dis. Act.Dis.
Baselines: ImageBind CAV-MAE DenseAV
Builds on: DenseAV CAV-MAE ImageBind DINO HuBERT
Closest papers by meaning
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.90)
- Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Language (CVPR 2024 · similarity 0.90)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.89)
- Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.89)
- Object-aware Sound Source Localization via Audio-Visual Scene Understanding (CVPR 2025 · similarity 0.88)
- SoundBrush: Sound as a Brush for Visual Scene Editing (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.