Research map / Vision-language and multimodal / Multimodal LLMs

ERGeoBench: A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ERGeoBench, a panorama-based embodied geo-localization benchmark with single-, panorama-, and embodied-view settings and four capability dimensions, evaluating nine MLLMs.

Problem

MLLMs are increasingly used as embodied agents, but global geo-localization is typically evaluated as passive single-image or one-shot panorama inference, lacking fine-grained, interactive evaluation of active evidence acquisition and spatial consistency.

Approach

Converts 2,207 global street-view panoramas into an egocentric embodied environment where agents control yaw, pitch and zoom to acquire views, and diagnoses performance across four capabilities (foundational perception, spatial awareness, common sense, geo-localization reasoning) with a composite Geo-localization Score (GLS).

Key results

• Gemini-2.5-Pro achieves the highest overall GLS (65.93 single, 68.28 panorama, 69.02 embodied) among evaluated models.
• Gemini-3-Flash reaches 22.96% Acc@1km in embodied view versus 7.62% for InternVL3-8B, while spatial awareness averages stay below 53 for all models.
• Foundational perception averages exceed 70 for most proprietary models, but cross-view common-sense and spatial scores are markedly lower (e.g., GPT-4o spatial avg 43.86 vs perception 82.78).

Surprising findings

• Top models in the embodied setting match or surpass their panorama-view GLS, suggesting active viewpoint selection can compensate for restricted initial observation.
• Gemini-2.0-Flash outperforms GPT-4o on spatial awareness (52.55 vs 43.86) despite being an older model, and GPT-5 gains substantially over GPT-4o on spatial awareness (64.90 vs 43.86).

Limitations

• Evaluation is in a simulated panorama-based environment without translational movement or physical deployment.
• Annotations are partly model-assisted (GPT-4o and a VLM checker), risking self-referential bias despite human verification.
• OOD wilderness test uses only 20 scenes.

Experimental setup

Datasets: OpenStreetMap Street-view panoramas (2,207 global)

Benchmarks: ERGeoBench

Metrics: GLS Acc@1km Acc@25km Acc@200km Acc@750km Acc@2500km city accuracy country accuracy median error average error

Baselines: IMAGEO-Bench GLOBE GRE Suite GAEA G3

Builds on: nuScenes

Closest papers by meaning

• OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding (NeurIPS 2025 · similarity 0.92)
• Recognition through Reasoning: Reinforcing Image Geo-localization with Large Vision-Language Models (NeurIPS 2025 · similarity 0.90)
• From Correspondence to Actions: Human-Like Multi-Image Spatial Reasoning in Multi-modal Large Language Models (ICML 2026 · similarity 0.90)
• Textual Supervision Enhances Geospatial Representations in Vision-Language Models (ICML 2026 · similarity 0.90)
• GRE Suite: Geo-localization Inference via Fine-Tuned Vision-Language Models and Enhanced Reasoning Chains (NeurIPS 2025 · similarity 0.89)
• NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.89)