Research map / Large language models / Agents, code and math
TANGO: Training-free Embodied AI Agents for Open-world Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TANGO, a training-free neuro-symbolic system where an LLM (GPT-4o) composes PointNav, memory-based exploration, and vision primitives via few-shot programs to solve open-set ObjectNav, GOAT lifelong navigation, and OpenEQA.
Problem
Extending LLM-based program composition (as in VisProg and ViperGPT for images) to embodied agents that must observe and act in 3D environments, handling multiple navigation and question-answering tasks without task-specific training.
Approach
An LLM planner with 15 in-context examples outputs executable pseudo-programs over a library of navigation and perception primitives built on a pre-trained PointGoal policy; the exploration policy of VLFM is extended with a per-pixel feature memory for multi-goal lifelong navigation.
Key results
- On HM3D-OVON val unseen, TANGO reaches 35.5% SR and 19.5% SPL, on par with VLFM (35.2/19.6) and below DAgRL+OD (37.1).
- On OpenEQA, TANGO scores 37.2 LLM-Match, second to Socratic LLMs with frame captions (38.1).
- On GOAT-Bench val unseen, TANGO achieves 32.1% SR, above the best prior modular GOAT (24.9), with SPL 16.5 vs 17.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TANGO instead of DAgRL+OD | HM3D-OVON object-goal navigation · HM3D-OVON val unseen | SR: 37.1 ±0.2 -> 35.5 ±0.3 | Hurt |
| Use TANGO instead of DAgRL+OD | HM3D-OVON object-goal navigation · HM3D-OVON val unseen | SPL: 19.9 ±0.3 -> 19.5 ±0.3 | Hurt |
| Use TANGO instead of the strongest reported baseline | OpenEQA question answering · OpenEQA | Score (LLM-Match): 38.1 ± 1.8 -> 37.2 ± 1.8 | Hurt |
| Use TANGO instead of SenseAct-NN Skill Chain | GOAT-Bench lifelong navigation · GOAT-Bench val unseen | SR: 29.5 -> 32.1 | Helped |
| Use TANGO instead of Modular GOAT | GOAT-Bench lifelong navigation · GOAT-Bench val unseen | SPL: 17.2 -> 16.5 | Hurt |
Surprising findings
- Only about 11.2% of EQA failures are attributed to incorrect LLM-generated code; most failures stem from prompt-related issues or the detect module.
Limitations
- May struggle with overly complex or ambiguous prompts that hinder primitive or target identification.
- Path efficiency drops significantly when the memorized target object is incorrect.
- Detector produces many false positives in synthetic 3D scenes.
Experimental setup
Datasets: HM3D-OVON GOAT-Bench OpenEQA
Benchmarks: HM3D-OVON GOAT-Bench OpenEQA
Metrics: SR SPL DTG LLM-Match score
Baselines: RL (OVON) BCRL DAgRL DAgRL+OD VLFM Human Agent Blind LLMs Socratic LLMs w/ Frame Captions SenseAct-NN Skill Chain Modular GOAT
Builds on: VisProg ViperGPT VLFM Neural Module Networks
Closest papers by meaning
- Towards Learning a Generalist Model for Embodied Navigation (CVPR 2024 · similarity 0.89)
- An Embodied Generalist Agent in 3D World (ICML 2024 · similarity 0.88)
- World Model Implanting for Test-time Adaptation of Embodied Agents (ICML 2025 · similarity 0.88)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.88)
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.88)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.