Research map / Multi-agent systems and games / Human-AI and goal-conditioned
CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAVEN, a conversational audio-visual navigation agent that learns via a budget-aware POSMDP and RL when to navigate by audio, query an oracle for instructions, or ask natural-language questions, trained with a new AVN-Instruct dataset.
Problem
Audio-goal navigation in noisy, sporadic sound settings is hard; prior audio-visual-language navigation (AVLEN) only allows uni-directional oracle instructions and no agent questions.
Approach
Model the agent's interaction as a budget-aware partially observable semi-Markov decision process with three options (audio-visual navigation, oracle-instruction navigation, question-answer navigation), where a learned selector policy decides when to ask and a differential penalty rewards correct questions; questions are generated from forecasted trajectories and panoramic views.
Key results
- Under unheard sounds, CAVEN with language feedback reaches 42.0% success vs. 26.2% for AVLEN with language feedback.
- With distractor sounds, CAVEN achieves 21.3% success vs. 14.0% for AVLEN and 11.8% for SAVi (no feedback).
- Ablation shows delta_ques=0 (no penalty on correct questions) gives 42.0% success vs. 32.1% at delta_ques=1.0 under unheard sounds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CAVEN's bidirectional conversational policy instead of AVLEN's language feedback | Audio-visual navigation · Unheard Sound | Success: 36.7 -> 42.0 | Helped |
| use CAVEN's bidirectional conversations instead of Model Uncertainty triggering | Audio-visual navigation · Distractor Sound | Success: 19.6 -> 21.3 | Helped |
| set δques to 0.0 rather than 0.5 (ablation) | CAVEN question policy · Unheard Sound | Success: 36.5 -> 42.0 | Helped |
| set δques to 0.0 rather than 1.0 (ablation) | CAVEN question policy · Unheard Sound | Success: 32.1 -> 42.0 | Helped |
| use Transformer (CLIP embedding) instead of Transformer (GloVe embedding) (ablation) | QuestionNet · Landmark RxR language generation | BLEU-4: 0.341 -> 0.367 | Helped |
| use a transformer with GloVe instead of a GRU with GloVe (ablation) | QuestionNet · Landmark RxR language generation | BLEU-4: 0.301 -> 0.341 | Helped |
| use egocentric occupancy maps instead of RGB+D inputs (ablation) | TrajectoryNet · Trajectory forecasting, 1 step | Accuracy: 76.7% -> 85.2% | Helped |
| use egocentric occupancy maps instead of RGB inputs (ablation) | TrajectoryNet · Trajectory forecasting, 1 step | Accuracy: 75.8% -> 85.2% | Helped |
| use CAVEN's QuestionNet and FollowerNet instead of Paul et al.'s module pretrained on Landmark RxR (ablation) | CAVEN language-based policy · Unheard Sound | Success: 26.4 -> 31.1 | Helped |
| learn the selector policy instead of using Model Uncertainty triggering (ablation) | CAVEN · Unheard Sound | Success: 33.3 -> 42.0 | Helped |
| add the separate language-based policy branch to the question branch with oracle instructions (ablation) | CAVEN · Unheard Sound | Success: 36.7 -> 42.0 | Helped |
| add the question-answer policy branch to the language-based policy setup (ablation) | CAVEN · Unheard Sound | Success: 31.1 -> 42.0 | Helped |
Surprising findings
- Not penalizing correct questions (delta=0) performs much better than penalizing all questions equally, suggesting the penalty gap acts as an incentive for better trajectory forecasting.
Limitations
- Interactions with the oracle may force the agent to wait for feedback.
- Relies on a budget limit on oracle instructions (e.g., 2 per episode) to keep oracle effort low.
- AVN-Instruct sub-instructions are synthesized by a pretrained QuestionNet, so quality depends on that generator.
Experimental setup
Datasets: SoundSpaces semantic audio-visual navigation dataset Landmark-RxR AVN-Instruct
Benchmarks: SoundSpaces semantic audio-goal navigation (heard, unheard, distractor sound)
Metrics: Success Rate SPL SNA DTG SWS SNI SNO BLEU-4 METEOR ROUGE-L
Baselines: Gan et al. Chen et al. AV-WaN SMT+Audio SAVi AVLEN Random Uniform Model Uncertainty
Builds on: AVLEN DD-PPO SAVi Landmark-RxR
Closest papers by meaning
- RAO-Nav: Probing Omni-Language Models for Zero-shot Semantic Audio-Visual Navigation (NeurIPS 2026 · similarity 0.91)
- RILA: Reflective and Imaginative Language Agent for Zero-Shot Semantic Audio-Visual Navigation (CVPR 2024 · similarity 0.89)
- Towards Audio-Visual Navigation in Noisy Environments: A Large-Scale Benchmark Dataset and an Architecture Considering Multiple Sound-Sources (AAAI 2025 · similarity 0.88)
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.87)
- Aux-Think: Exploring Reasoning Strategies for Data-Efficient Vision-Language Navigation (NeurIPS 2025 · similarity 0.87)
- Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.