Research map / Vision-language and multimodal / Robot action and navigation
RAO-Nav: Probing Omni-Language Models for Zero-shot Semantic Audio-Visual Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAO-Nav, a zero-shot semantic audio-visual navigation pipeline that uses one frozen omni-language model (Qwen2.5-Omni-7B) for audio goal prediction, global reasoning and action, plus a test-time latent navigation reasoning (LNR) module.
Problem
Semantic audio-visual navigation requires an embodied agent to identify a sounding target and reach it in unseen environments, but specialized models need heavy task-specific training and multi-LLM systems are complex to deploy.
Approach
Use a single frozen omni-language model for the whole SAVN loop (audio classification, waypoint selection via an external localization expert, geometric turn-then-forward control), and elicit better reasoning at test time by optimizing latent tokens with a likelihood-based intrinsic reward to select target-relevant RGB history.
Key results
- RAO-Nav with LNR on Qwen2.5-Omni-7B reaches 39.2% SR on unheard-sound Matterport3D test, above the best supervised baseline K-SAVEN (34.4%) and zero-shot RILA (35.4%), without training data.
- Removing the OLM-predicted audio category from the Localization Expert drops SR from 39.2% to 12.6%.
- Pure VLMs with oracle audio info reach at most 14.7% SR (Qwen3-VL-30B), far below RAO-Nav.
Surprising findings
- Adding depth observations lowers SR from 39.2% to 28.6%, which the authors attribute to extra reasoning burden on the OLM.
- Zero-shot RAO-Nav has lower SPL than K-SAVEN (20.7% vs 23.4%) and DTG is higher, so trajectories are less efficient despite higher SR.
Limitations
- Failures are mainly due to inaccurate sound source localization, which depends on the external Localization Expert.
- Zero-shot methods still lag supervised models in SPL, SWS and DTG, indicating less efficient trajectories.
- The Global Navigation Instruction setting uses small sample sizes (164/172/248) and OLMs perform poorly on open-set audio and option matching.
Experimental setup
Datasets: SAVi benchmark (SoundSpaces, Matterport3D) Global Navigation Instruction set (164 closed-set and 172 open-set audio samples, 248 navigation samples)
Benchmarks: SAVi SoundSpaces Matterport3D unheard-sound test (1,000 episodes) Global Navigation Instruction setting
Metrics: SR SPL SNA DTG SWS accuracy
Baselines: AudioGoal AV-WAN SAVi AVLEN K-SAVEN RILA LLaVA-NeXT Qwen2.5-VL-7B Qwen3-VL-8B Qwen3-VL-30B Baichuan-Omni-7B
Builds on: SAVi RILA SoundSpaces Habitat
Closest papers by meaning
- RILA: Reflective and Imaginative Language Agent for Zero-Shot Semantic Audio-Visual Navigation (CVPR 2024 · similarity 0.92)
- CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments (AAAI 2024 · similarity 0.91)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.90)
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.90)
- Towards Audio-Visual Navigation in Noisy Environments: A Large-Scale Benchmark Dataset and an Architecture Considering Multiple Sound-Sources (AAAI 2025 · similarity 0.89)
- FantasyVLN: Unified Multimodal Chain-of-Thought Reasoning for Vision-and-Language Navigation (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.