Research map / Vision-language and multimodal / Cross-modal learning
Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks humans and multimodal AI models on sound source localization under audio-visual conflict and absence, and proposes EchoPin, a HRTF- and cochleagram-based stereo model trained on the new AudioCOCO dataset.
Problem
Multimodal AI models for sound source localization are not well understood under cross-modal conflicts or absent cues, and they show a bias toward vision, unlike humans who prioritize audio.
Approach
Render stereo audio for COCO images using a depth-aware Unity 3D simulation with HRTF filtering and cochleagram inputs, then train a dual-encoder contrastive model (EchoPin) on the resulting AudioCOCO dataset, and compare against humans in six audio-visual conditions.
Key results
- EchoPin raises congruent A-Acc from 11.2% (small objects) to 76.0% (large objects), outperforming IS3 across object sizes.
- Under ConflictVCue, IS3 drops to near-chance while humans and EchoPin remain significantly above chance.
- Humans show horizontal localization accuracy of 86.1% vs 66.7% vertical; EchoPin shows a similar horizontal advantage without human data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EchoPin instead of IS3 | SSL localization model · MultiInstLoc Size2 | A-Acc (%): 7.9 -> 24.1 | Helped |
| use EchoPin instead of IS3 | SSL localization model · MultiInstLoc Size3 | A-Acc (%): 22.4 -> 47.1 | Helped |
| use stereo audio rather than averaging channels to mono (ablation) | EchoPin · Overall Size1 | A-Acc (%): 3.6 -> 9.7 | Helped |
| use stereo audio rather than averaging channels to mono (ablation) | EchoPin · Overall Size2 | A-Acc (%): 15.8 -> 31.3 | Helped |
| use stereo audio rather than averaging channels to mono (ablation) | EchoPin · Overall Size3 | A-Acc (%): 31.4 -> 47.6 | Helped |
| use cochleagrams instead of standard mel-spectrograms with stereo audio (ablation) | EchoPin · Overall Size1 | A-Acc (%): 5.3 -> 9.7 | Helped |
| use cochleagrams instead of standard mel-spectrograms with stereo audio (ablation) | EchoPin · Overall Size2 | A-Acc (%): 17.0 -> 31.3 | Helped |
| use cochleagrams instead of standard mel-spectrograms with stereo audio (ablation) | EchoPin · Overall Size3 | A-Acc (%): 35.2 -> 47.6 | Helped |
| rotate the interaural axis by 90 degrees to simulate vertically aligned ears (ablation) | EchoPin-Ro · auditory horizontal-versus-vertical localization | localization precision asymmetry | Mixed |
| train EchoPin-S on VGG-SS and Flickr-SoundNet | EchoPin-S · VGG-SS and Flickr-SoundNet | CIoU: 42.96 -> 43.61; 84.40 -> 85.25 | Helped |
Surprising findings
- Gaussian noise audio lowers vision-only V-Acc relative to silent audio for all models, suggesting noisy audio reduces over-reliance on vision.
- EchoPin is more easily misled by conflicting visual cues than humans, even though it is stable when visual cues are absent.
Limitations
- Single static images and single-source localization; no temporal dynamics from video.
- Simulated audio does not fully capture real-world acoustics such as refraction.
- Models remain inferior to humans in complex scenes, with visual saliency bias persisting.
Experimental setup
Datasets: MSCOCO 2014 VGGSound AudioCOCO FlickrSoundNet VGG-Sound VGG-SS AVSBench
Benchmarks: AudioCOCO test set VGG-SS Flickr-SoundNet
Metrics: A-Acc V-Acc chance-corrected gain cIoU
Baselines: IS3 CAVP AVSegformer ImageBind LanguageBind SSLTI LVS FNAC
Builds on: IS3 PyCochleagram AudioCOCO
Closest papers by meaning
- EgoSound: Benchmarking Sound Understanding in Egocentric Videos (CVPR 2026 · similarity 0.90)
- STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence (ICLR 2026 · similarity 0.88)
- Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-visual LLMs (NeurIPS 2026 · similarity 0.88)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.88)
- Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling (CVPR 2024 · similarity 0.88)
- Object-aware Sound Source Localization via Audio-Visual Scene Understanding (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.