Research map / Vision-language and multimodal / Multimodal LLMs
UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal Models in Multi-View Urban Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces UrBench, a 11.6K-question multi-view (street, satellite, cross-view) benchmark across 14 tasks in 4 dimensions for evaluating LMMs in urban scenarios, and evaluates 21 LMMs against humans.
Problem
Existing benchmarks for large multimodal models in urban settings focus on region-level tasks from single views, leaving role-level tasks and cross-view understanding of urban environments under-evaluated.
Approach
A benchmark combining region-level and role-level urban tasks across street-view, satellite-view and cross-view imagery, with instance-level cross-view annotations created by a detection-matching method (Grounding DINO boxes, ray-traced projection, IoU matching) and questions built via LMM-based, rule-based, and human-based generation.
Key results
- GPT-4o achieves 61.2% overall accuracy on UrBench, versus 69.9% for human experts (average gap of 17.4%).
- Most LMMs perform near random (about 26-28%) on Camera Localization and Orientation; cross-view tasks average 36.2% across models.
- VILA-1.5-40B is the best open-source model at 53.1% overall accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use human experts rather than GPT-4o | Evaluation on UrBench · UrBench Overall | accuracy: 61.2 -> 69.9; 17.4% | Helped |
| Evaluate on cross-view tasks rather than street-view tasks | LMMs · UrBench views | average accuracy: 54.6% -> 36.2% | Hurt |
| Evaluate on satellite-view tasks rather than street-view tasks | LMMs · UrBench views | average accuracy: 54.6% -> 42.3% | Hurt |
| Use street-view or cross-view images instead of satellite-view images | LMMs on City Retrieval · City Retrieval (CR) | average accuracy: 50.9% -> 62.9%; 50.9% -> 63.8% | Mixed |
| Use multi-image inputs rather than concatenated single-image inputs | LLaVA-NeXT-Interleave-7B vs LLaVA-NeXT-8B · UrBench Overall | accuracy: 38.8; 40.4 | No clear effect |
Surprising findings
- Multi-image models show little advantage over single-image counterparts, suggesting comprehension rather than multi-image processing is the bottleneck.
- LMMs perform better on street-view than satellite-view and cross-view tasks, and inconsistently across views of the same geolocation.
Limitations
- Most open-source models evaluated are small (7-8B scale) and results depend on zero-shot prompting protocols.
- Cross-view annotations rely on automatic detection-matching with human verification, which may introduce bias.
- City Retrieval favors LMMs' geographic world knowledge, making human comparison less direct.
Experimental setup
Datasets: Cityscapes Mapillary Traffic Sign Dataset VIGOR IM2GPS OpenStreetMap Google Street View Google Earth
Benchmarks: UrBench
Metrics: accuracy
Baselines: Human Random Gemini-1.5-Flash Claude-3.5-Sonnet InternVL2 LLaVA-NeXT VILA-1.5
Builds on: MMMU MuirBench Grounding DINO
Closest papers by meaning
- MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models (ICLR 2025 · similarity 0.89)
- UrbanFeel：A Comprehensive Benchmark for Temporal and Perceptual Understanding of City Scenes through Human Perspective (ICLR 2026 · similarity 0.89)
- ERGeoBench: A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models (ICML 2026 · similarity 0.88)
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.88)
- MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence (ICLR 2026 · similarity 0.88)
- OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.