Research map / Vision-language and multimodal / Multimodal LLMs

TraveLLaMA: A Multimodal Travel Assistant with Large-Scale Dataset and Structured Reasoning

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces TravelQA (265k travel QA pairs incl. 5k expert CoT examples) and TraveLLaMA, a Vicuna-13B LLaVA-1.5-style VLM fine-tuned with Travel-CoT reasoning plus a ReAct-style travel agent evaluated via a 500-user SUS study.

Problem

Existing multimodal models lack specialized travel knowledge and contextual understanding of urban environments, and no large multimodal travel QA dataset exists for training travel assistants.

Approach

Builds a travel-specific multimodal QA dataset (text and map/street-view QA generated with GPT-4 from POI sources) and adds a structured spatial/temporal/practical chain-of-thought reasoning stage to fine-tune VLMs, wrapped in a ReAct-style agent with tool calls.

Key results

• TraveLLaMA (Vicuna-13B, 336px) reaches 77.8 full score vs 76.0 for fine-tuned LLaVA-1.5 Vicuna-13B, reported as +10.8% over the pretrained baseline.
• Fine-tuning on TravelQA improves all listed VLMs by 6.2-9.4% (e.g., Qwen-VL full 68.1 to 74.5).
• A 500-participant SUS study gives the TraveLLaMA agent 82.5 vs 76.3 for Claude 3.5.

Surprising findings

• The reported 10.8% gain over the pretrained baseline is measured against the pretrained LLaVA-1.5 baseline, while the gain over the fine-tuned LLaVA-1.5 is only about 1.8 points, which the abstract does not make clear.

Limitations

• Travel-CoT is described inconsistently (two-stage formulation vs joint training); the CoT dataset is only 5k examples.
• Test set is built from the same GPT-4-generated pipeline, and MCQ conversion with distractors may not reflect free-form quality.
• The agent comparison is against a single general model (Claude 3.5) with no details on task design or statistics.

Experimental setup

Datasets: TravelQA

Benchmarks: TravelQA

Metrics: accuracy (four-choice MCQ) SUS score weighted full score

Baselines: BLIP-2 InstructBLIP Shikra Qwen-VL Qwen-VL-Chat LLaVA-1.5 Claude 3.5

Builds on: LLaVA-1.5 Qwen-VL Shikra ReAct TravelPlanner

Closest papers by meaning

• IMAIA: Interactive Maps AI Assistant for Travel Planning and Geo-Spatial Intelligence (CVPR 2026 · similarity 0.86)
• TravelPlanner: A Benchmark for Real-World Planning with Language Agents (ICML 2024 · similarity 0.86)
• MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models (ICML 2025 · similarity 0.86)
• FRIEDA: Benchmarking Multi-Step Cartographic Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.85)
• SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence (CVPR 2026 · similarity 0.85)
• ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Maps (CVPR 2026 · similarity 0.85)