Research map / Vision-language and multimodal / Multimodal LLMs
GTR-Bench: Evaluating Geo-Temporal Reasoning in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GTR-Bench, a 420-question benchmark for geo-temporal reasoning that combines map images and multi-camera videos (CityFlow outdoor, MTMMC indoor), and finds that top VLMs score far below humans.
Problem
Existing spatial-temporal benchmarks cover either egocentric video/image reasoning or static geographic reasoning on maps, but not geographic spatial-temporal reasoning about moving targets across non-overlapping cameras in a camera network that requires integrating maps and multi-view video.
Approach
A benchmark of seven geo-temporal tasks (three basic, four combinatorial) built from real multi-camera tracking datasets, where each question pairs a map with video clips from multiple cameras, scored by MCQ accuracy or a novel ST-IoU metric combining camera correctness with temporal IoU.
Key results
- Gemini-2.5-Pro achieves the best average score of 34.93% on GTR-Bench, far below human performance of 78.61%.
- Best open-source model InternVL3-38B reaches 30.76% average, while a traditional ReID baseline gets 45.72% average MCQ accuracy.
- Models score higher on MCQ spatial location questions than on ST-IoU spatial-temporal forecasting; e.g., GPT-4o has 53.33 MCQ vs 20.53 ST-IoU on outdoor NSF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add temporal-spatial map+video reasoning with graphical map context for Gemini-2.5-Pro versus other models | Proprietary VLMs · GTR-Bench average | Average score | No clear effect |
| Replace graphic map with textual map description (ablation) | Gemini-Pro · GTR-Outdoor MS/TF/MTTF | accuracy: TF 13.16 to 0.00 (image vs text), MTTF 19.18 to 1.11 | Hurt |
| Replace graphic map with textual map description (ablation) | Gemini-Pro and InternVL-38B · GTR-Indoor GL and CR | accuracy: GL 26.67 to 73.33 for Gemini-Pro; CR 70.00 to 83.33 | Helped |
| Use an ReID-based traditional baseline instead of VLM reasoning | ReID Baseline · GTR-Bench average | average MCQ accuracy: 45.72 vs Gemini-2.5-Pro 34.93 | Helped |
| Scale model size within InternVL3 family (2B to 38B) | InternVL3 · GTR-Bench average | average score: 18.46 (2B) to 22.10 (8B) to 30.76 (38B) | Helped |
| Evaluate with spatial-correct-conditioned ST-IoU versus standard (gap between MCQ and ST-IoU) | Proprietary and open models · GTR-Outdoor NSF | MCQ accuracy vs ST-IoU: GPT-4o 53.33 MCQ vs 20.53 ST-IoU | Mixed |
Surprising findings
- A traditional ReID baseline (45.72% average) outperforms all evaluated VLMs, including the best proprietary model.
- Gemini-2.5-Pro performs better on indoor than outdoor for several tasks, contrary to expectations.
Limitations
- Benchmark has only 420 questions (60 per task), a relatively small evaluation set.
- Frame sampling capped at 20 frames across multiple videos to fit model input constraints.
- Context utilization analysis relies on an LLM judge (Qwen-max) assessing reasoning text.
Experimental setup
Datasets: CityFlow MTMMC
Benchmarks: GTR-Bench
Metrics: MCQ accuracy ST-IoU Soft ST-IoU
Baselines: Claude-3.7-Sonnet Claude-4-Sonnet GPT-4o GPT-5 Gemini-2.5-Pro InternVL3-2B InternVL3-8B InternVL3-38B Qwen2-VL-2B-Instruct Qwen2-VL-7B-Instruct Qwen2.5-VL-7B-Instruct Qwen2.5-VL-32B-Instruct
Builds on: CityFlow MTMMC ReasonMap STI-Bench VSI-Bench
Closest papers by meaning
- TimeSpot: Benchmarking Geo-Temporal Understanding in Vision–Language Models in Real-World Settings (ICML 2026 · similarity 0.92)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence (ICLR 2026 · similarity 0.91)
- Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.91)
- OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding (NeurIPS 2025 · similarity 0.91)
- From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs (CVPR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.