Research map / Vision-language and multimodal / Robot action and navigation
GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GOAT-Bench, an open-vocabulary, multi-modal, lifelong navigation benchmark (category, language, image goals in sequences of 5-10 subtasks on HM3DSem) and benchmarks modular and RL navigation methods with and without memory.
Problem
Existing embodied navigation benchmarks handle only one goal modality, a closed set of few object categories, and episodic settings, so universal navigation agents that take category, language, or image goals and leverage past experience in the same environment are not evaluated.
Approach
Chain open-vocabulary goals of three modalities (category name, LLM-generated language description, image) into lifelong episodes of 5-10 subtasks per scene, where the agent's memory carries over between subtasks, and benchmark modular (semantic/instance map) and end-to-end RL policies on it.
Key results
- SenseAct-NN Skill Chain achieves 29.2% SR on VAL SEEN versus 26.3% for Modular GOAT, but lower SPL (12.8 vs 17.5).
- Modular GOAT with ground-truth semantics reaches 56.7% SR and 40.3 SPL on VAL SEEN, showing perception is a major bottleneck.
- Dropping memory between subtasks roughly halves Modular GOAT SPL (17.6 to 9.4) while SenseAct-NN Monolithic barely changes (9.4 to 9.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SenseAct-NN Skill Chain instead of Modular GOAT | GOAT navigation · VAL SEEN | SR (%): 26.3 -> 29.2 | Helped |
| use SenseAct-NN Skill Chain instead of Modular GOAT | GOAT navigation · VAL SEEN | SPL (%): 17.5 -> 12.8 | Hurt |
| add an explicit semantic and instance map (ablation) | Modular GOAT · VAL SEEN | SPL: 9.4 -> 17.6 | Helped |
| add an explicit semantic and instance map (ablation) | Modular GOAT · VAL SEEN | SR: 21.2 -> 26.4 | Helped |
| carry the hidden state across subtasks (ablation) | SenseAct-NN Monolithic · VAL SEEN | SPL: 9.0 -> 9.4 | Helped |
| carry the hidden state across subtasks (ablation) | SenseAct-NN Monolithic · VAL SEEN | SR: 14.9 -> 16.8 | Helped |
| maintain instance-specific memory and use image keypoints for matching | Modular GOAT · GOAT-Bench validation splits | SPL (%) | Helped |
| use CLIP embeddings for language goals | GOAT navigation methods · Language goals | success rate | Hurt |
| use CroCo-v2 features instead of CLIP features for image-goal policies | SenseAct-NN Skill Chain · Image goals | success rate | Helped |
| retain memory across subtasks | Modular GOAT · later subtasks in GOAT episodes | SPL: 12.4 to 18.7 | Helped |
| retain memory across subtasks | SenseAct-NN Monolithic · later subtasks in GOAT episodes | SPL: 5.6 to 10.6 | Helped |
| replace object category names with synonyms | Modular GOAT · VAL SEEN object goals | success rate | Hurt |
Surprising findings
- The SenseAct-NN monolithic policy with CLIP goal encoders performs poorly on language and image goals, which the authors attribute to CLIP's limited instance-specific features.
- SenseAct-NN Skill Chain is the most robust to goal noise (about 25% average drop) while Modular GOAT is the least robust (53% drop).
Limitations
- CLIP features poorly capture instance-specific attributes for language and image goals.
- Modular methods suffer from perception bottlenecks such as detector failure on synonyms.
- Evaluation is in simulation only (HM3DSem in Habitat) and the authors note real-world evaluation is time-intensive.
Experimental setup
Datasets: HM3DSem HM3D-OVON InstanceImageNav-HM3D
Benchmarks: GOAT-Bench VAL SEEN GOAT-Bench VAL SEEN SYNONYMS GOAT-Bench VAL UNSEEN
Metrics: success rate SPL
Baselines: Modular GOAT Modular CLIP on Wheels SenseAct-NN Skill Chain SenseAct-NN Monolithic GOAT-GTSem
Builds on: HM3DSem HM3D-OVON CLIP CroCo-v2 VER Modular GOAT
Closest papers by meaning
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.87)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.87)
- Run, Ruminate, and Regulate: A Dual-process Thinking System for Vision-and-Language Navigation (AAAI 2026 · similarity 0.86)
- LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation (NeurIPS 2026 · similarity 0.86)
- TANGO: Training-free Embodied AI Agents for Open-world Tasks (CVPR 2025 · similarity 0.86)
- 3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.