Research map / Vision-language and multimodal / Human and embodied modeling
UrbanVLP: Multi-Granularity Vision-Language Pretraining for Urban Socioeconomic Indicator Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UrbanVLP, a vision-language pretraining framework that fuses satellite and street-view imagery with LMM-generated, calibrated text (scored by a new PerceptionScore) to predict urban socioeconomic indicators.
Problem
Urban socioeconomic indicator prediction from imagery suffers from single-granularity (satellite-only) visual information and from hallucinated, homogenized text in prior LLM-based pipelines (UrbanCLIP).
Approach
Combines satellite and street-view image features with location embeddings under dual-branch contrastive losses (global image-text and token-level fine-grained alignment), and generates street-view captions with ShareGPT4V using segmentation-ratio prompts, filtered by a reference-free PerceptionScore (CLIPScore plus SDXL-based CycleScore).
Key results
- UrbanVLP beats the best baseline StructuralUrban in R^2 on all six Beijing indicators (e.g., +3.3% Carbon, +2.5% GDP, +4.9% Night Light), with an average R^2 gain of about 3.95% across cities.
- Average PerceptionScore of generated text rises from 0.675 (UrbanCLIP) to 0.714 (UrbanVLP); mean text similarity drops from 0.28 to 0.19.
- In transfer across 16 source-target city pairs for carbon prediction, UrbanVLP averages R^2 0.574 versus 0.495 for UrbanCLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| integrate satellite and street-view imagery with calibrated text and location features | UrbanVLP · Beijing Carbon, Population, GDP, Night Light, House Price, and POI | R²: 3.3%, 2.3%, 2.5%, 4.9%, 1.0% and 2.4% | Helped |
| add street-view branch to the satellite branch (ablation) | UrbanVLP · CityView-Beijing | R²: 3.56% | Helped |
| add textual information during pretraining rather than using visual features without textual augmentation (ablation) | UrbanVLP · CityView-Beijing Carbon, Population, and GDP | R²: 0.724 -> 0.769; 0.686 -> 0.714; 0.501 -> 0.537 | Helped |
| add location encoding branch (ablation) | UrbanVLP · CityView-Beijing | R²: 1.5% | Helped |
| calibrate generated descriptions by filtering low PerceptionScore texts (ablation) | UrbanVLP · CityView-Beijing Carbon, Population, and GDP | R²: 0.724 -> 0.769; 0.686 -> 0.714; 0.501 -> 0.537 | Helped |
| replace addition fusion with concatenation (ablation) | UrbanVLP · CityView-Beijing Carbon, Population, and GDP | R²: 0.759 vs 0.769; 0.705 vs 0.714; 0.526 vs 0.537 | Hurt |
| replace addition fusion with MLP fusion (ablation) | UrbanVLP · CityView-Beijing Carbon, Population, and GDP | R²: 0.761 vs 0.769; 0.710 vs 0.714; 0.532 vs 0.537 | Hurt |
| use satellite and street-view imagery together instead of street-view imagery alone | UrbanVLP · CityView urban indicator prediction | R² | Helped |
| transfer visual encoders trained on source cities and fine-tune on target cities | UrbanVLP vs UrbanCLIP · Carbon, 16 source-target city pairs | R²: 0.495 -> 0.574 | Helped |
| increase the maximum street-view images per region during training (ablation) | UrbanVLP · CityView | R² | Helped |
| set loss combination weights to α=0.5 and β=0.5 (ablation) | UrbanVLP · CityView | R² | Helped |
| generate descriptions with ShareGPT4V rather than the other evaluated open-source LMMs | UrbanVLP text generation · Generated descriptions | PerceptionScore: 0.714 | Helped |
Surprising findings
- Street-view-only imagery (UrbanCLIP-SV) underperforms satellite-only UrbanCLIP, showing street-view lacks macro context on its own.
- InstructBLIP was excluded because adding geospatial coordinates to its prompts significantly degraded generated text quality.
Limitations
- Text generation relies on an open-source LMM and LLM-judged quality checks (GPT-4V) rather than human evaluation.
- Dataset covers only four Chinese cities and six indicators.
- Fusion method ablation shows differences among fusion options are minor.
Experimental setup
Datasets: CityView (self-constructed satellite, street-view, and text data) ODIAC carbon emissions WorldPop population night light data Anjuke house prices POI data
Benchmarks: CityView Beijing CityView Shanghai CityView Guangzhou CityView Shenzhen
Metrics: R^2 RMSE MAE PerceptionScore FPS
Baselines: ViT PG-SimCLR UrbanCLIP UrbanCLIP-SV Vision-LSTM StructuralUrban
Builds on: UrbanCLIP CLIP GeoCLIP ShareGPT4V
Closest papers by meaning
- CityLens: Evaluating Large Vision-Language Models for Urban Socioeconomic Sensing (ICLR 2026 · similarity 0.91)
- UrbanDataLayer: A Unified Data Pipeline for Urban Science (NeurIPS 2024 · similarity 0.87)
- UrbanFeel：A Comprehensive Benchmark for Temporal and Perceptual Understanding of City Scenes through Human Perspective (ICLR 2026 · similarity 0.86)
- DynamicVL: Benchmarking Multimodal Large Language Models for Dynamic City Understanding (NeurIPS 2025 · similarity 0.86)
- Urban Region Embedding via Multi-View Contrastive Prediction (AAAI 2024 · similarity 0.84)
- GeoLLM: Extracting Geospatial Knowledge from Large Language Models (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.