Research map / Large language models / Agents, code and math
FLAME: Learning to Navigate with Multimodal LLM in Urban Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds FLAME, an Otter/OpenFlamingo-based multimodal LLM agent for urban vision-and-language navigation, trained with a three-phase tuning scheme on GPT-4-synthesized captions, route summaries, and rationales.
Problem
Text-only and multimodal LLMs underperform specialized models on urban vision-and-language navigation (VLN), and MLLMs have not been effectively adapted to outdoor navigation with efficient handling of many observations.
Approach
Adapt a Flamingo-style MLLM to urban navigation using strided cross-attention over per-observation visual tokens (prioritizing recent views), and a three-phase tuning curriculum: street-view captioning, route summarization plus imitation, then end-to-end navigation training, with GPT-4/GPT-4V synthesized supervision.
Key results
- On Touchdown test, FLAME reaches 40.20 TC, a 7.3% gain over the prior SOTA Loc4Plan (32.90 TC).
- On Map2seq test, FLAME reaches 52.44 TC, a 3.74% gain over VELMA (48.70 TC).
- Ablation: adding phase 1 and phase 2 tuning raises Touchdown dev TC from 40.16 to 41.28 and Map2seq dev TC from 53.57 to 56.95.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FLAME instead of Loc4Plan | Urban VLN navigation · Touchdown Test Set | TC: +7.3% | Helped |
| Add P1 single-perception tuning (ablation) | FLAME · Touchdown Dev Set | TC: 40.16 -> 41.15 | Helped |
| Add P1 single-perception tuning (ablation) | FLAME · Map2seq Dev Set | TC: 53.57 -> 54.69 | Helped |
| Add P2 multiple-perception tuning after P1 (ablation) | FLAME · Touchdown | TC: 41.15 -> 41.28 | Helped |
| Add P2 multiple-perception tuning after P1 (ablation) | FLAME · Map2seq | TC: 54.69 -> 56.95 | Helped |
| Prioritize recent observations using smaller strided cross-attention stride (ablation) | FLAME · Touchdown and Map2seq | TC | Helped |
| Increase decoding paths to 8 and use temperature 1.0 rather than greedy decoding (ablation) | FLAME reasoning · Touchdown Test Set subset | TC: +1.72% | Helped |
| Increase decoding paths to 8 and use temperature 1.0 rather than greedy decoding (ablation) | FLAME reasoning · Touchdown Test Set subset | RC: +1.75% | Helped |
| Increase decoding paths to 8 rather than greedy decoding (ablation) | FLAME reasoning · Map2seq Test Set subset | TC: +3.91% | Helped |
| Increase decoding paths to 8 rather than greedy decoding (ablation) | FLAME reasoning · Map2seq Test Set subset | RC: +3.62% | Helped |
Surprising findings
- Baseline methods drop sharply when restricted to a narrow, non-panoramic field of view (e.g., RCONCAT* and GA* on Touchdown fall to roughly 6-8 TC), while FLAME keeps working with that input.
- Increasing stride size generally lowers task completion, suggesting that attending to many past observations overwhelms the agent.
Limitations
- Relies on GPT-4 and GPT-4V to synthesize captions, route summaries, and rationales, and metric evaluation uses GPT-4 with a manual calibration step.
- Reasoning evaluation is limited to the subset of instances with synthetic rationales (about 6,500 of 9,326 Touchdown pairs), and human checks cover only 50 instances per dataset.
- Evaluated only on two urban datasets (Touchdown and Map2seq) built on StreetLearn.
Experimental setup
Datasets: Touchdown Map2seq StreetLearn OpenStreetMap
Benchmarks: Touchdown Map2seq
Metrics: Task Completion SPD nDTW Rationale Coherence Rationale-Action Alignment
Baselines: ORAR Loc4Plan VELMA VLN-Video GA RCONCAT VLN-Trans ARC+L2S
Builds on: Flamingo OpenFlamingo Otter ReAct VELMA
Closest papers by meaning
- VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View (AAAI 2024 · similarity 0.90)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.87)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.87)
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.86)
- Towards Learning a Generalist Model for Embodied Navigation (CVPR 2024 · similarity 0.86)
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.