Research map / Video / Video-language understanding
VIR-Bench: Evaluating Geospatial and Temporal Understanding of MLLMs via Travel Video Itinerary Reconstruction
Hao Wang, Eiki Murata, Lingfang Zhang, Ayako Sato, So Fukuda, Ziqi Yin, Wentao Hu, Keisuke Nakao, Yusuke Nakamura, Sebastian Zwirner, Yi-Chia Chen, Hiroyuki Otomo and 2 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase input frames from 64 to 128 (ablation) | GPT-4.1 · POI node prediction | F1: 39.6 -> 52.9 | Improved |
| increase input frames from 64 to 256 (ablation) | GPT-4.1 · transition edge prediction | F1: 27.6 -> 34.4 | Improved |
| increase input frames from 128 to 256 (ablation) | GPT-4.1 · inclusion edge prediction | F1: 78.8 -> 76.5 | Mixed |
| raise reasoning effort from low to high (ablation) | o4-mini · transition edge prediction | F1: 30.0 -> 43.8 | Improved |
| raise reasoning effort from low to high (ablation) | o4-mini · prefecture node prediction | F1: 86.8 -> 86.4 | No clear effect |
| add audio input (ablation) | Gemini-2.5-Flash · transition edge prediction | F1: 22.3 -> 41.7 | Improved |
| add audio input (ablation) | Gemini-2.5-Flash · inclusion edge prediction | F1: 82.6 -> 78.5 | Hurt |
| add audio input (ablation) | Gemini-2.5-Flash · prefecture node prediction | F1: 82.5 -> 85.1 | Improved |
| scale Qwen2.5-VL from 7B to 72B | Qwen2.5-VL · transition edge prediction | F1: 1.3 -> 20.8 | Improved |
| scale LLaVA-Video from 7B to 72B | LLaVA-Video · POI node prediction | F1: 6.0 -> 7.8 | No clear effect |
| give agent both POI list and video instead of POI only | travel-planning agent (Gemini-2.5-Pro) · Attraction (1-5) | mean score: 3.58 -> 3.73 | Improved |
| use video only instead of POI list for planning agent | travel-planning agent (Gemini-2.5-Pro) · Alignment with video | share of plans unrelated: 31% completely unrelated | Hurt |
Closest papers by meaning
- GTR-Bench: Evaluating Geo-Temporal Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
- From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs (CVPR 2026 · similarity 0.88)
- TraveLLaMA: A Multimodal Travel Assistant with Large-Scale Dataset and Structured Reasoning (AAAI 2026 · similarity 0.88)
- OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding (NeurIPS 2025 · similarity 0.87)
- TimeSpot: Benchmarking Geo-Temporal Understanding in Vision–Language Models in Real-World Settings (ICML 2026 · similarity 0.87)
- ERGeoBench: A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
