Research map / Vision-language and multimodal / Human and embodied modeling
GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
GeoLink pretrains a ViT-L remote sensing foundation model jointly with a heterogeneous GNN encoder over OpenStreetMap vector data, using masked reconstruction, region-image contrastive alignment, and object-patch spatial consistency, then fuses RS and OSM features for unimodal and multimodal geographic tasks.
Problem
Remote sensing foundation models mostly use imagery alone, and existing ways of using OpenStreetMap data are indirect and lose spatial information due to the modality gap between raster imagery and OSM vector data.
Approach
Model OSM as a heterogeneous graph (points, polylines, polygons with topological edges) encoded by a GATConv-based GNN, align it with RS images via region-level contrastive learning, and fuse fine-grained object-patch features with sinusoidal position embeddings in a two-way cross-attention fusion encoder; RS and OSM are both masked during pretraining for efficiency.
Key results
- GeoLink achieves the best kNN accuracy on most of seven RS classification benchmarks, e.g., 87.33% on RESISC-45 kNN versus 85.42% for Scale-MAE, and 98.19% on UCMerced kNN.
- On UFZ segmentation, multimodal GeoLink reaches 60.00 mIoU versus 55.40 unimodal; POP r² rises from 49.76 (unimodal) to 51.88 and CO2 r² from 62.37 to 65.12 with OSM fusion.
- Ablation shows the 60-epoch GeoLink matches or beats MAE trained 300 epochs (kNN 87.06 vs 85.97) at roughly 1/4 of the cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse OSM with RS features at downstream task time | GeoLink · Urban village identification | mIoU: 79.53 -> 81.68 | Helped |
| Fuse OSM with RS features at downstream task time | GeoLink · POP estimation | r^2: 49.76 -> 51.88 | Helped |
| Fuse OSM with RS features at downstream task time | GeoLink · CO2 estimation | r^2: 62.37 -> 65.12 | Helped |
| Add region-image contrastive and spatial consistency losses (ablation) | GeoLink pretraining · Classification (7-benchmark mean) | kNN accuracy: 78.12 -> 87.06 | Helped |
| Add region-image contrastive loss (ablation) | GeoLink pretraining · Classification (7-benchmark mean) | kNN accuracy: 86.12 -> 87.06 | Helped |
| Add spatial consistency loss (ablation) | GeoLink pretraining · UFZ segmentation (multimodal) | mIoU: 57.39 -> 60.00 | Helped |
| Reduce RS masking ratio to 50% (ablation) | GeoLink pretraining · Classification (7-benchmark mean) | kNN accuracy: 87.06 -> 83.21 | Hurt |
| Increase RS masking ratio to 80% (ablation) | GeoLink pretraining · Classification (7-benchmark mean) | kNN accuracy: 87.06 -> 85.21 | Hurt |
| Increase RS masking ratio to 90% (ablation) | GeoLink pretraining · Classification (7-benchmark mean) | kNN accuracy: 87.06 -> 78.02 | Hurt |
| Remove fusion position embeddings (ablation) | GeoLink · UFZ segmentation (multimodal) | mIoU: 60.00 -> 56.22 | Hurt |
| Replace GAT with GCN in the OSM encoder (ablation) | GeoLink · Classification (7-benchmark mean) | kNN accuracy: 87.06 -> 87.01 | Hurt |
| Replace GNN-based OSM encoder with two Transformer blocks (ablation) | GeoLink · Classification (7-benchmark mean) | kNN accuracy: 87.06 -> 86.14 | Hurt |
Surprising findings
- Using only polyline/polygon centroids as position proxies gives nearly the same performance as the sampled key-point design.
- Averaging more sinusoidal key-point embeddings degrades performance, which the authors attribute to over-smoothing of the positional signal.
Limitations
- Supports only RGB imagery and cannot process multispectral RS data.
- Sinusoidal position embedding for OSM vectors can lose spatial detail.
- Baselines are not modality-aligned, since no existing RS FM uses both RS and OSM; multimodal baselines were adapted with zero-padding to RGB.
Experimental setup
Datasets: SkyScript-top30 OpenStreetMap (via Overpass API) MLRSNet EuroSAT WHU-RS19 OPTIMAL-31 RESISC-45 AiRound UCMerced Five-Billion-Pixels AI4SmallFarms xView2
Benchmarks: MLRSNet EuroSAT WHU-RS19 OPTIMAL-31 RESISC-45 AiRound UCMerced Five-Billion-Pixels AI4SmallFarms xView2 SpaceNet7 UFZ segmentation benchmark
Metrics: top-1 accuracy mIoU r^2
Baselines: GASSL MMEarth Scale-MAE Cross-Scale MAE CROMA DOFA
Builds on: MAE FLIP InfoNCE GATConv SkyScript
Closest papers by meaning
- Bridging Remote Sensors with Multisensor Geospatial Foundation Models (CVPR 2024 · similarity 0.88)
- SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (CVPR 2024 · similarity 0.88)
- MMM-RS: A Multi-modal, Multi-GSD, Multi-scene Remote Sensing Dataset and Benchmark for Text-to-Image Generation (NeurIPS 2024 · similarity 0.87)
- GeoViS: Geospatially Rewarded Visual Search for Remote Sensing Visual Grounding (CVPR 2026 · similarity 0.86)
- No One Knows the State-of-the-Art in Geospatial Foundation Models (NeurIPS 2026 · similarity 0.86)
- GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.