Research map / Vision-language and multimodal / Multimodal LLMs

FRIEDA: Benchmarking Multi-Step Cartographic Reasoning in Vision-Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces FRIEDA, a 500-question benchmark of multi-map, multi-step cartographic reasoning over real document maps, showing that 11 LVLMs reach at most ~38% accuracy versus 84.87% for humans.

Problem

Existing map VQA benchmarks treat maps like charts or cover only narrow spatial relations and single maps, so LVLM cartographic reasoning (topological, metric, directional relations across multiple maps) remains under-evaluated.

Approach

A benchmark of real maps from public reports across 32 countries, with questions spanning border, equal, intersect, within, distance, and orientation relations, free-form answers, multi-map reasoning, and a contextual setting requiring retrieval of the relevant map among distractors; scored with LLM-as-judge, MAPE, and angular tolerance.

Key results

• Gemini-2.5-Pro achieves the top overall accuracy of 38.20% on FRIEDA-direct, and GPT-5-Think 37.20%, versus 84.87% human average.
• Best open-source model Ovis2.5-9B-Think reaches 24.80% (25.80% in the direct table), with no clear relation between model size and performance.
• Contextual setting yields nearly unchanged accuracy (Gemini 36.60%, GPT-5-Think 37.00%), with 88.03% per-question agreement between direct and contextual settings for open-source models.

Surprising findings

• Retrieval among distractor maps barely changes accuracy, suggesting retrieval is not the main bottleneck.
• Ovis2.5-9B-Think (9B) outperforms many models with 70B+ parameters, indicating scale is not decisive.

Limitations

• Only English-language documents, causing regional and language representation bias.
• Domain skew toward planning, environmental, and government reports.
• Small benchmark of 500 questions, with some spatial-relation subsets under 100 items.

Experimental setup

Datasets: FRIEDA

Benchmarks: FRIEDA

Metrics: accuracy MAPE

Baselines: Gemini-2.5-Pro GPT-5-Think Claude-Sonnet-4 Qwen2.5-VL-72B InternVL3-78B Ovis2.5-9B-Think

Builds on: MapQA ReMI DocVQA

Closest papers by meaning

• MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models (ICML 2025 · similarity 0.90)
• ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Maps (CVPR 2026 · similarity 0.89)
• SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence (CVPR 2026 · similarity 0.88)
• Vision-Language Reasoning for Geolocalization: A Reinforcement Learning Approach (AAAI 2026 · similarity 0.88)
• SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning (NeurIPS 2025 · similarity 0.87)
• Recognition through Reasoning: Reinforcing Image Geo-localization with Large Vision-Language Models (NeurIPS 2025 · similarity 0.87)