Research map / Vision-language and multimodal / Multimodal LLMs

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Probes ridge regressions on layer-wise residuals of ViT, MAE, DINOv2, CLIP, LLaVA, Qwen2.5-VL and Gemma 3 to show textual supervision yields stronger latitude/longitude representations, and that prompts and dimension swapping can shift them.

Problem

Whether vision-only and vision-language models implicitly encode geospatial (latitude/longitude) information from training without explicit geospatial supervision, and how training objectives and scale affect it.

Approach

Layer-wise linear (ridge) probing of [CLS]/last-token residuals across vision-only, contrastive VLM, and multimodal LLM families on localizability-clustered image sets, plus prompting and top-dimension swapping experiments to test whether geospatial information is causally steerable.

Key results

• Text-trained VLMs reach average probe R^2 above 0.4 for larger models, while vision-only models are mostly below 0.3; DINOv2-giant is outperformed by CLIP-base on average.
• Prompting with a lat/lon query raises Qwen probe R^2 up to 0.88 on the Landmarks dataset.
• Keeping about 40% of top-ranked dimensions recovers nearly maximal R^2 across models; swapping 50% of dimensions moved a Step Pyramid image's predicted location from Egypt to Rome.

Surprising findings

• Gemma's probe R^2 turns negative in later layers without a textual prompt.
• On matched MetaCLIP data, MetaCLIP-large (0.70) outperforms the 7B Web-SSL DINO model (0.56).

Limitations

• Probing uses pretrained models with unknown training data, so memorization and data differences may confound results.
• Analysis limited to Landmarks and YFCC clusters; no satellite imagery studied.
• Generation steering becomes unstable for longer outputs.

Experimental setup

Datasets: YFCC100M (4.2M subset, 5k sampled) Google Landmarks V2 (580k subset, 5k sampled)

Benchmarks: YFCC100M clusters Google Landmarks subset country identification fine-tuning

Metrics: R^2 test accuracy SSI geo-bias median geodesic error median proximity error

Baselines: ViT ViT-MAE DINOv2-giant Web-SSL DINO MetaCLIP

Builds on: CLIP DINOv2 LLaVA-1.5 GeoCLIP PIGEON

Closest papers by meaning

• GeoDiT: A Diffusion-based Vision-Language Model for Geospatial Understanding (CVPR 2026 · similarity 0.90)
• ERGeoBench: A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models (ICML 2026 · similarity 0.90)
• Recognition through Reasoning: Reinforcing Image Geo-localization with Large Vision-Language Models (NeurIPS 2025 · similarity 0.90)
• Vision-Language Reasoning for Geolocalization: A Reinforcement Learning Approach (AAAI 2026 · similarity 0.90)
• Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.89)
• Stitch and Tell: A Structured Data Augmentation Method for Spatial Understanding (NeurIPS 2025 · similarity 0.89)