Research map / Large language models / Agents, code and math
MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MapEval, a 700-question multiple-choice benchmark testing foundation models on textual, API-based (ReAct agent with cached Google Maps tools) and visual map reasoning, finding that top models trail human performance by 20+ points.
Problem
Foundation models' geo-spatial and map-based reasoning (distances, directions, routing, POI search, trip planning, map images) has not been rigorously evaluated.
Approach
A three-mode MCQ benchmark built from Google Maps data (textual context via MapQaTor, API access through a cached pseudo-Google Maps environment, and map snapshots) that evaluates LLMs, VLMs and ReAct agents on the same map-based question types, with unanswerable and commonsense items.
Key results
- Claude-3.5-Sonnet reaches 66.33% on MapEval-Textual and 61.65% on MapEval-Visual (Claude-3-5-Sonnet), while human performance is 86.67% (textual) and 82.23% (visual).
- On MapEval-API, Claude-3.5-Sonnet agent achieves 64.00% overall, far above the best open-source agent Llama-3.2-90B at 39.67%.
- Adding a calculator tool raises Claude-3.5-Sonnet straight-line distance accuracy from 51.06% to 85.11%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a calculator (ablation) | Claude-3.5-Sonnet · MapEval-Textual straight-line distance questions | accuracy: 51.06% to 85.11% | Helped |
| add a calculator (ablation) | GPT-4o-mini · MapEval-Textual cardinal direction questions | accuracy: 29.17% to 91.67% | Helped |
| add a calculator (ablation) | Gemma-2.0-9B · MapEval-Textual straight-line distance questions | accuracy: 29.79% to 68.90% | Helped |
| replace ReAct with Chameleon (ablation) | GPT-3.5-Turbo · MapEval-API | accuracy: 27.33 to 49.33 | Helped |
| provide map API tools instead of pre-fetched textual context | Claude-3.5-Sonnet · Nearby | accuracy: 73.49 to 55.42 | Hurt |
| provide map API tools instead of pre-fetched textual context | Claude-3.5-Sonnet · Routing | accuracy: 75.76 to 65.15 | Hurt |
| use map API tools instead of pre-fetched textual context | MapEval models · Trip | accuracy: approximately 22% | Helped |
Surprising findings
- Gemini-1.5-Pro achieves 85% on unanswerable textual questions while most models score 0-45%.
- Open-source Qwen2.5-VL-72B reaches 60.35% on visual tasks, near the closed-source leaders.
Limitations
- Covers only five Google Maps APIs (Text Search, Place Details, Nearby Search, Directions, Distance Matrix), excluding other map categories.
- Cached API environment means results may not reflect live map data and become outdated as APIs change.
- Uses multiple-choice format and does not test prompt-design variations.
Experimental setup
Datasets: MapEval
Benchmarks: MapEval-Textual MapEval-API MapEval-Visual
Metrics: accuracy
Baselines: Claude-3.5-Sonnet GPT-4o Gemini-1.5-Pro GPT-4-Turbo Gemini-1.5-Flash GPT-4o-mini GPT-3.5-Turbo Llama-3.1-70B Llama-3.2-90B Qwen2.5-72B Qwen2.5-VL-72B ReAct
Builds on: ReAct MMLU MapQaTor Chameleon Travel Planner
Closest papers by meaning
- FRIEDA: Benchmarking Multi-Step Cartographic Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
- SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence (CVPR 2026 · similarity 0.90)
- Vision-Language Reasoning for Geolocalization: A Reinforcement Learning Approach (AAAI 2026 · similarity 0.89)
- Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark (AAAI 2024 · similarity 0.89)
- GRE Suite: Geo-localization Inference via Fine-Tuned Vision-Language Models and Enhanced Reasoning Chains (NeurIPS 2025 · similarity 0.89)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.