Research map / Vision-language and multimodal / Cross-modal learning
Binaural SoundNet: Predicting Semantics, Depth and Motion With Binaural Sounds
Dengxin Dai, Arun Balajee Vasudevan, Jiří Matas, Luc Van Gool
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use binaural instead of mono audio | Spectrogram encoder · Auditory semantic prediction (spectrogram, manual test) | mIoU (%): 21.14 -> 28.02 | Improved |
| add S3R auxiliary task (ablation) | Ours(B) · Auditory semantic prediction (spectrogram, manual test) | mIoU (%): 28.02 -> 37.80 | Improved |
| add depth auxiliary task (ablation) | Ours(B) · Auditory semantic prediction (spectrogram, manual test) | mIoU (%): 28.02 -> 36.11 | Improved |
| add motion auxiliary task (ablation) | Ours(B) · Auditory semantic prediction (spectrogram, manual test) | mIoU (%): 28.02 -> 35.12 | Improved |
| add all three auxiliary tasks (S3R, depth, motion) (ablation) | Ours(B) · Auditory semantic prediction (spectrogram, manual test) | mIoU (%): 28.02 -> 38.27 | Improved |
| add all three auxiliary tasks (ablation) | Ours(B), DDSP encoder · Auditory semantic prediction (DDSP, manual test) | mIoU (%): 20.67 -> 25.44 | Improved |
| add all three auxiliary tasks (ablation) | Ours(B), spectrogram + DDSP encoder · Auditory semantic prediction (spectrogram + DDSP, manual test) | mIoU (%): 32.27 -> 42.16 | Improved |
| add ASPP module to audio encoder (ablation) | Mono audio encoder · Auditory semantic prediction | mIoU: 13.21 to 22.12 | Improved |
| increase input channels from mono to all 8 microphones | Ours(B) · Auditory semantic prediction | mIoU (%): 22.12% -> 40.32% | Improved |
| add auxiliary tasks (semantic, depth, motion) (ablation) | Binaural depth model, spectrogram · Auditory depth prediction | Abs Rel: 108.59 -> 83.22 | Improved |
| add auxiliary tasks (S3R, semantic, depth) (ablation) | Binaural motion model, spectrogram · Auditory motion prediction | EPE: 16.22 -> 14.04 | Improved |
| add semantic, depth and motion tasks to S3R (ablation) | S3R, spectrogram · Spatial sound super-resolution | MSE-1: 0.1228 -> 0.0940 | Improved |
Closest papers by meaning
- Cyclic Learning for Binaural Audio Generation and Localization (CVPR 2024 · similarity 0.90)
- Supervising Sound Localization by In-the-wild Egomotion (CVPR 2025 · similarity 0.89)
- OWL : Geometry-Aware Spatial Reasoning for Audio Large Language Models (ICLR 2026 · similarity 0.89)
- DeepASA: An Object-Oriented Multi-Purpose Network for Auditory Scene Analysis (NeurIPS 2025 · similarity 0.88)
- Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization (NeurIPS 2025 · similarity 0.88)
- Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-visual LLMs (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
