Research map / Vision-language and multimodal / Robot action and navigation
RILA: Reflective and Imaginative Language Agent for Zero-Shot Semantic Audio-Visual Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RILA, a zero-shot semantic audio-visual navigation agent that uses a frozen GPT-3.5 reflective planner plus an LLM imaginative assistant for room-layout reasoning, without training on navigation demonstrations.
Problem
Semantic audio-visual navigation (SAVN) requires an agent to find a sounding target object in unseen 3D scenes from intermittent audio, but existing methods need extensive RL training and generalize poorly.
Approach
Use a frozen LLM planner that treats audio-derived goal descriptions as possibly wrong and reflectively explores, combined with an LLM-based imaginative assistant that infers room layouts and gives strategic region-level advice, with perception modules converting audio and vision into text.
Key results
- RILA reaches 35.4% SR and 20.4% SWS on the SAVN test set without training demonstrations, above the zero-shot ESC baseline (23.6% SR) and above supervised SAVi (24.8% SR).
- Removing the Imaginative Assistant drops SR from 35.4% to 31.4%, and SWS from 20.4% to 15.3%.
- With ground-truth visual perception added, RILA reaches 62.0% SR on the validation set, indicating LLM planning is strong when perception is accurate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ImaAssistant to RILA (ablation) | RILA · SoundSpaces Matterport3D test | SR (%): 31.4 -> 35.4 | Helped |
| Add ImaAssistant to RILA (ablation) | RILA · SoundSpaces Matterport3D test | SPL (%): 9.6 -> 11.8 | Helped |
| Add ImaAssistant to RILA (ablation) | RILA · SoundSpaces Matterport3D test | SWS (%): 15.3 -> 20.4 | Helped |
| Replace GPT-3.5 frontier selection with random frontier selection (ablation) | RILA · SoundSpaces Matterport3D test | SR: 60.8 -> 19.8 | Hurt |
| Replace GPT-3.5 frontier selection with nearest-frontier selection (ablation) | RILA · SoundSpaces Matterport3D test | SR: 60.8 -> 9.8 | Hurt |
| Replace GPT-3.5 with Llama-2 7B for frontier selection (ablation) | RILA · SoundSpaces Matterport3D test | SR: 60.8 -> 39.4 | Hurt |
| Add ground-truth audio semantics (ablation) | RILA · SoundSpaces validation | SR: 30.2 -> 30.2 | No clear effect |
| Add ground-truth audio localization (ablation) | RILA · SoundSpaces validation | SR: 30.2 -> 52.4 | Helped |
| Add ground-truth visual perception (ablation) | RILA with ground-truth audio semantics and localization · SoundSpaces validation | SR: 52.4 -> 62.0 | Helped |
| Use RILA instead of K-SAVEN | SoundSpaces Matterport3D test · SoundSpaces Matterport3D test | SR (%): 34.4 -> 35.4 | Helped |
| Use RILA instead of K-SAVEN | SoundSpaces Matterport3D test · SoundSpaces Matterport3D test | SWS (%): 14.3 -> 20.4 | Helped |
Surprising findings
- Audio direction classification accuracy is only 73.7%, and ITD does not clearly indicate sound direction, so sound localization is identified as the main bottleneck.
- Replacing GPT-3.5 with Llama-2 7B in the planner drops SR to 39.4%, indicating weak spatial reasoning in the open model.
Limitations
- SPL is relatively low because the agent explores holistically and cautiously, traversing longer paths.
- Audio direction estimation is unreliable (73.7% accuracy), and error accumulates over steps.
- Fails when target objects are visually similar and goal localization is imprecise.
Experimental setup
Datasets: SoundSpaces Matterport3D
Benchmarks: SoundSpaces Matterport3D SAVN test set (1,000 episodes, 10 unseen scenes, 21 goal objects)
Metrics: SR SPL SNA DTG SWS
Baselines: AudioGoal AV-WAN SAVi AVLEN K-SAVEN Imagebind-LLM ESC Llama-2 7B
Builds on: ESC K-SAVEN SAVi GroundingDINO ImageBind-LLM
Closest papers by meaning
- RAO-Nav: Probing Omni-Language Models for Zero-shot Semantic Audio-Visual Navigation (NeurIPS 2026 · similarity 0.92)
- CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments (AAAI 2024 · similarity 0.89)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.87)
- ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination (ICLR 2025 · similarity 0.87)
- Planning from Imagination: Episodic Simulation and Episodic Memory for Vision-and-Language Navigation (AAAI 2025 · similarity 0.87)
- Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.