Research map / Vision-language and multimodal / Cross-modal learning
Towards Audio-Visual Navigation in Noisy Environments: A Large-Scale Benchmark Dataset and an Architecture Considering Multiple Sound-Sources
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BeDAViN, a 10.8-hour audio dataset for audio-visual navigation with multiple and noisy sound sources, and ENMuS3, a navigation agent with a sound event descriptor and multi-scale scene memory transformer.
Problem
Audio-visual navigation is studied mostly for single sound-source scenes; existing datasets have too few audio clips to simulate diverse multi-source environments, and existing frameworks degrade badly with multiple sources and background noise.
Approach
Builds a large binaural audio corpus (2,258 clips, 24 sound event and noise categories) for simulation and proposes ENMuS3, which estimates active sound-source categories and DoAs via a sound event descriptor and tracks the target with a multi-scale scene memory transformer (convolutional multi-resolution memory plus cross-attention decoder).
Key results
- ENMuS3 reaches 79.3% SR in single-source, 46.5% in multi-source, and 18.0% in noisy scenarios on BeDAViN, versus 66.3%, 37.9%, 14.1% for SMT + Audio (best baseline).
- BeDAViN contains 2,258 audio clips totaling 10.8 hours, about 33 times the duration of the SAVi audio dataset (1,157 seconds).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ENMuS3 instead of SMT + Audio | Audio-visual navigation · Single-source scenarios | SR: 66.3 -> 79.3 | Helped |
| Use ENMuS3 instead of SMT + Audio | Audio-visual navigation · Single-source scenarios | SPL: 31.9 -> 44.1 | Helped |
| Use ENMuS3 instead of SMT + Audio | Audio-visual navigation · Single-source scenarios | SNA: 47.3 -> 64.4 | Helped |
| Use ENMuS3 instead of SMT + Audio | Audio-visual navigation · Single-source scenarios | DTG: 3.9 -> 2.1 | Helped |
| Use ENMuS3 instead of SMT + Audio | Audio-visual navigation · Multi-source scenarios | SR: 37.9 -> 46.5 | Helped |
| Use ENMuS3 instead of SMT + Audio | Audio-visual navigation · Multi-source scenarios | SPL: 17.7 -> 24.0 | Helped |
| Use ENMuS3 instead of SMT + Audio | Audio-visual navigation · Multi-source scenarios | SNA: 24.5 -> 35.0 | Helped |
| Use ENMuS3 instead of SMT + Audio | Audio-visual navigation · Multi-source scenarios | DTG: 6.9 -> 6.0 | Helped |
| Use ENMuS3 instead of SMT + Audio | Audio-visual navigation · Noisy scenarios | SR: 14.1 -> 18.0 | Helped |
| Use ENMuS3 instead of SMT + Audio | Audio-visual navigation · Noisy scenarios | SPL: 6.1 -> 8.8 | Helped |
| Use ENMuS3 instead of SMT + Audio | Audio-visual navigation · Noisy scenarios | SNA: 9.4 -> 12.3 | Helped |
| Use ENMuS3 instead of SMT + Audio | Audio-visual navigation · Noisy scenarios | DTG: 10.1 -> 9.9 | Helped |
Surprising findings
- All methods have very low success rates in noisy scenarios, with the best at 18.0% SR.
Limitations
- Evaluation is only in simulation (SoundSpaces on Matterport3D); real-world deployment is future work.
- Noisy-scenario success rates remain low (18.0% SR for ENMuS3).
- Gym equipment category was omitted from the dataset.
Experimental setup
Datasets: BeDAViN SAVi-dataset AudioSet FSD50K freesound.org Matterport3D
Benchmarks: BeDAViN
Metrics: SR SPL SNA DTG
Baselines: Random Goal Follower ObjectGoal Av-Nav SAVi SMT + Audio
Builds on: SMT AV-Nav SAVi SoundSpaces
Closest papers by meaning
- RAO-Nav: Probing Omni-Language Models for Zero-shot Semantic Audio-Visual Navigation (NeurIPS 2026 · similarity 0.89)
- Object-aware Sound Source Localization via Audio-Visual Scene Understanding (CVPR 2025 · similarity 0.88)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.88)
- CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments (AAAI 2024 · similarity 0.88)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.87)
- Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.