Research map / Vision-language and multimodal / Multimodal LLMs

ENC-Bench: A Benchmark for Evaluating Multimodal Large Language Models in Electronic Navigational Chart Understanding

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ENC-Bench, a 20,490-sample benchmark built from 840 NOAA S-57 electronic navigational charts that evaluates 10 MLLMs on perception, spatial reasoning, and maritime decision-making under a zero-shot protocol; the best model reaches 47.88%.

Problem

It is unclear whether multimodal LLMs can reliably interpret safety-critical Electronic Navigational Charts, which use standardized vector symbology, scale-dependent rendering, and multi-constraint geometry that general chart and map benchmarks do not cover.

Approach

A hierarchical benchmark (perception, spatial reasoning, decision-making) generated from raw IHO S-57 vector data via a calibrated vector-to-image pipeline with automated checks and expert review, with systematic variation across lighting modes and scale levels.

Key results

• Best model Gemini-2.5-Pro reaches 47.88% average accuracy across perception and decision-making tasks, vs. 29.76% random chance.
• Geographic coordinate localization underperforms direct pixel localization for most models (e.g., Gemini-2.5-Pro 17.36% Geo vs 21.43% pixel at 200px).
• Distance measurement accuracy at 20% tolerance peaks at about 26% (Gemini-2.5-Flash 25.93%), and Anchorage Selection reaches at most 30.50%.

Surprising findings

• Spatial reasoning performance improves under night mode rendering, even though perception and decision-making degrade.
• Geographic coordinate prediction underperforms direct pixel prediction despite being theoretically more precise.

Limitations

• Evaluation is zero-shot only; no fine-tuning or training-based methods are explored.
• Decision-making tasks use procedurally simulated scenarios and synthetic overlays (e.g., red triangle vessel symbol), which may not match real navigation conditions.
• Error analysis covers only 200 sampled incorrect predictions from one model.

Experimental setup

Datasets: NOAA ENC charts (S-57)

Benchmarks: ENC-Bench

Metrics: accuracy Acc@T mean error

Baselines: GPT-4o Gemini-2.5-Flash Qwen3-VL-235B-Instruct Qwen3-VL-32B-Instruct InternVL-3-38B GLM-4.5V Llama-4-Maverick-17B

Builds on: IHO S-57 OpenCPN GDAL

Closest papers by meaning

• ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding (CVPR 2026 · similarity 0.83)
• ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.83)
• CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs (NeurIPS 2024 · similarity 0.83)
• FRIEDA: Benchmarking Multi-Step Cartographic Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.83)
• Is your VLM Sky-Ready? A Comprehensive Spatial Intelligence Benchmark for UAV Navigation (CVPR 2026 · similarity 0.82)
• Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.82)