Research map / Multi-agent systems and games / Human-AI and goal-conditioned
VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VELMA, an LLM agent for urban Street View navigation that verbalizes trajectories and CLIP-scored landmark observations into a text prompt, used with few-shot prompting and LoRA finetuning with response-based learning.
Problem
Vision-and-language navigation in large-scale real-world urban environments (Street View) requires grounding long, human-written instructions in visual observations over ~40-step trajectories, which prior LLM agents could not handle.
Approach
Embodies a text-only LLM by verbalizing the trajectory and visual observations: an LLM extracts landmarks from instructions, CLIP with a standardized score determines landmark visibility per panorama view, and a template verbalizer turns these into observation text appended to the prompt for next-word action prediction. The paper also modifies the Touchdown environment's forward/rotation semantics
Key results
- VELMA-GPT-4 with 2-shot prompting reaches 10.0 TC on Touchdown and 23.1 TC on Map2seq (dev set).
- VELMA-RBL finetuned LLaMA-7b reaches 26.4 TC on Touchdown and 47.5 TC on Map2seq (test set), about 77% and 57% relative gains over prior SOTA.
- Vision ablation: OpenCLIP landmark scoring improves Touchdown TC from 15.0 (no image) to 23.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with response-based learning instead of teacher-forcing alone (ablation) | VELMA-FT · Touchdown development | TC: +2.9 | Helped |
| Train with response-based learning instead of teacher-forcing alone (ablation) | VELMA-FT · Map2seq development | TC: +7.5 | Helped |
| Add OpenCLIP landmark scoring instead of omitting landmark observations (ablation) | VELMA · Touchdown development | TC: 15.0 -> 23.4 | Helped |
| Add OpenCLIP landmark scoring instead of omitting landmark observations (ablation) | VELMA · Map2seq development | TC: 37.8 -> 41.3 | Helped |
| Use OpenCLIP instead of CLIP in the landmark scorer (ablation) | VELMA · Touchdown development | TC: 20.8 -> 23.4 | Helped |
| Use OpenCLIP instead of CLIP in the landmark scorer (ablation) | VELMA · Map2seq development | TC: 39.2 -> 41.3 | Helped |
Surprising findings
- Blind text-only LLM agents with landmark verbalization reach substantial task completion without any image features, with GPT-4 invoking turn_around usefully.
- OPT models barely outperform a forward-only heuristic even at 65B parameters.
Limitations
- GPT models evaluated only on development sets due to cost and data leakage concerns.
- Few-shot TC remains far below finetuned ORAR-style models on Touchdown and Map2seq.
- Landmark visibility threshold tau was set by inspection, not tuned systematically.
Experimental setup
Datasets: Touchdown Map2seq
Benchmarks: Touchdown Map2seq
Metrics: task completion shortest-path distance key point accuracy
Baselines: ORAR-ResNet ORAR-OpenCLIP forward-only heuristic VLN Transformer
Builds on: ORAR CLIP LoRA imitation learning
Closest papers by meaning
- FLAME: Learning to Navigate with Multimodal LLM in Urban Environments (AAAI 2025 · similarity 0.90)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.88)
- Run, Ruminate, and Regulate: A Dual-process Thinking System for Vision-and-Language Navigation (AAAI 2026 · similarity 0.88)
- Towards Learning a Generalist Model for Embodied Navigation (CVPR 2024 · similarity 0.87)
- FantasyVLN: Unified Multimodal Chain-of-Thought Reasoning for Vision-and-Language Navigation (CVPR 2026 · similarity 0.87)
- ERGeoBench: A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.