Research map / Vision-language and multimodal / Multimodal LLMs

CityLens: Evaluating Large Vision-Language Models for Urban Socioeconomic Sensing

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CityLens, a benchmark evaluating 17 LVLMs on predicting 11 urban socioeconomic indicators from satellite and street view imagery under three paradigms (direct, normalized, feature-based regression).

Problem

Existing urban socioeconomic sensing work lacks a unified, broad benchmark for assessing how large vision-language models predict socioeconomic indicators from satellite and street-view imagery across regions, tasks, and modalities.

Approach

A multi-city, multi-domain region-level benchmark that pairs satellite and street view imagery with ground-truth indicators and evaluates LVLMs with three paradigms: direct metric prediction, normalized 0-9.9 estimation, and LASSO regression on LVLM-extracted visual attribute scores.

Key results

• Feature-based regression with LVLM-extracted 13-attribute features outperforms direct and normalized prediction; best mean R^2 is 0.338 (Gemma3-27B) among LVLMs versus 0.417 for UrbanVLP.
• Fine-tuned Qwen2.5-VL-7B reaches R^2 0.872 on Building Height and 0.628 on GDP and Drive Ratio, well above zero-shot LVLMs.
• Mental Health and Bachelor Ratio show low R^2 for most LVLMs, with some near zero (e.g., 0.001 for Qwen2.5VL-3B on Bachelor Ratio).

Surprising findings

• Larger models do not consistently beat smaller ones; Gemma3-12B beats Gemma3-27B on GDP and Life Expectancy.
• Street view images alone perform comparably to combined street view and satellite input, while satellite alone performs significantly worse.

Limitations

• Ground-truth labels are unavailable for many cities, especially in the Global South, causing uneven task coverage and geographic bias.
• Google/Baidu street view images cannot be redistributed due to licensing; only pano IDs and scripts are released.
• Gemini models limited to 10 images, so 9 street views were used for them instead of 10.

Experimental setup

Datasets: CityLens (Google/Baidu street view + Esri satellite) CityLens-Mapillary GDP 1km global dataset WorldPop Zillow Home Value Index Local Data for Better Health SafeGraph

Benchmarks: CityLens CityLens-Mapillary

Metrics: R^2 nRMSE

Baselines: UrbanCLIP UrbanVLP GPT-4o-mini Gemini-2.0-Flash Gemini-2.5-Flash Llama4-Maverick Qwen2.5-VL-32B

Builds on: GeoLLM UrbanCLIP UrbanVLP

Closest papers by meaning

• UrbanVLP: Multi-Granularity Vision-Language Pretraining for Urban Socioeconomic Indicator Prediction (AAAI 2025 · similarity 0.91)
• UrbanFeel：A Comprehensive Benchmark for Temporal and Perceptual Understanding of City Scenes through Human Perspective (ICLR 2026 · similarity 0.89)
• UrbanDataLayer: A Unified Data Pipeline for Urban Science (NeurIPS 2024 · similarity 0.87)
• DynamicVL: Benchmarking Multimodal Large Language Models for Dynamic City Understanding (NeurIPS 2025 · similarity 0.87)
• CityPulse: Fine-Grained Assessment of Urban Change with Street View Time Series (AAAI 2024 · similarity 0.85)
• GeoLLM: Extracting Geospatial Knowledge from Large Language Models (ICLR 2024 · similarity 0.84)