Research map / Vision-language and multimodal / Cross-modal learning
Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DenseAV, a dual-encoder trained by contrastive learning on videos whose multi-head dense similarity aggregator yields high-resolution audio-visual local features that localize words and sounds without supervision.
Problem
Global audio-visual contrastive models (e.g., ImageBind, CAVMAE) learn strong retrieval embeddings but their local features do not align across modalities, so they cannot localize spoken words or sounds in images.
Approach
A multi-head dense similarity aggregator (max over heads and visual space, average over audio time) applied to local DINO and HuBERT features, with a cross-term disentanglement regularizer that lets heads separate language from general sound.
Key results
- Speech prompted segmentation mAP 48.7% vs 32.2% for DAVENet (prior best), a ~51% relative increase.
- Cross-modal retrieval on AudioSet Acc@10 68.2% (I→A) vs 64.5% for ImageBind, using fewer than half the trainable parameters.
- Two-head model achieves 99.9% predictive and 91.2% activation disentanglement between sound and language heads.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DenseAV instead of DAVENet | speech-prompted semantic segmentation · Speech semantic segmentation | mAP: 32.2% -> 48.7% | Helped |
| use DenseAV instead of CAVMAE | sound-prompted semantic segmentation · Sound semantic segmentation | mAP: 26.0% -> 32.4% | Helped |
| use DenseAV instead of the strongest reported baseline | cross-modal retrieval · PlacesAudio retrieval | I → A Acc. @10: 81.7% -> 94.2% | Helped |
| use DenseAV instead of the strongest reported baseline | cross-modal retrieval · AudioSet retrieval | I → A Acc. @10: 64.5% -> 68.2% | Helped |
| remove disentanglement loss and head max-pooling (ablation) | DenseAV · Sound/language disentanglement | Pred. Dis.: 64.1% -> 99.9% | Hurt |
| remove disentanglement loss (ablation) | DenseAV with head max-pooling · Sound/language disentanglement | Act. Dis.: 86.5% -> 91.2% | Hurt |
| replace multi-head feature aggregation with average pooling (ablation) | DenseAV · Speech semantic segmentation | mAP: 20.1% -> 48.2% | Hurt |
| replace multi-head feature aggregation with a CLS token (ablation) | DenseAV · Speech semantic segmentation | mAP: 20.6% -> 48.2% | Hurt |
| replace multi-head feature aggregation with SimPool (ablation) | DenseAV · Speech semantic segmentation | mAP: 35.3% -> 48.2% | Hurt |
| replace multi-head feature aggregation with average pooling (ablation) | DenseAV · PlacesAudio retrieval | V → A: 92.0% vs 93.5% | No clear effect |
| replace multi-head feature aggregation with a CLS token (ablation) | DenseAV · PlacesAudio retrieval | A → V: 89.8% vs 93.8% | No clear effect |
Surprising findings
- ImageBind scores only 1.10% Acc@10 on PlacesAudio retrieval, indicating it cannot recognize speech.
- DINO outperforms CLIP as visual backbone because of better-behaved local tokens.
Limitations
- Speech and sound segmentation evaluated on a constructed subset (3030 pairs; 106 for sound), limited in scale.
- Two-head disentanglement analysis is specific to K=2 and training on both AudioSet and PlacesAudio.
Experimental setup
Datasets: AudioSet PlacesAudio ADE20K VGGSound LibriSpeech
Benchmarks: ADE20K speech prompted segmentation ADE20K sound prompted segmentation Places Audio cross-modal retrieval AudioSet cross-modal retrieval
Metrics: mAP mIoU Accuracy@10 PredDis ActDis
Baselines: DAVENet CAVMAE ImageBind SimPool Average Pool CLS Token
Builds on: InfoNCE MISA DAVENet DINO HuBERT LoRA
Closest papers by meaning
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.90)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.89)
- Dense Audio-Visual Event Localization Under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration (AAAI 2025 · similarity 0.88)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.87)
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.87)
- Audio-Visual Segmentation via Unlabeled Frame Exploitation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.