Research map / Vision-language and multimodal / CLIP and zero-shot
Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a remote-sensing image encoder to align with frozen CLIP image embeddings of co-located Flickr ground photos (no text labels), yielding zero-shot satellite VLM for classification, retrieval, segmentation, and VQA.
Problem
Building open-vocabulary vision-language models for satellite imagery is hampered by the lack of large-scale text annotations for remote sensing images.
Approach
Use geotagged internet ground photos as an intermediary: train a satellite image encoder to match the frozen CLIP image embeddings of co-located ground images with a multi-positive contrastive loss, so text alignment transfers transitively through CLIP's text encoder; a patch-level variant aligns ViT patch features to ground images for localization.
Key results
- GRAFT ViT-B/16 reaches 63.76% zero-shot EuroSAT accuracy vs 53.59% for CLIP ViT-B/16 (about 10 points).
- On NAIP, GRAFT ViT-B/16 gets 49.38% per-class accuracy on Satlas NAIP segmentation vs 26.99% for CLIPSeg.
- With ViperGPT, GRAFT raises average zero-shot RSVQA accuracy from 37.69% (GLIP) to 44.05%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| align satellite imagery to co-located ground-image CLIP embeddings without text annotations | GRAFT ViT-B/16 · EuroSAT | accuracy: 53.59 -> 63.76 | Helped |
| align satellite imagery to co-located ground-image CLIP embeddings without text annotations | GRAFT ViT-B/16 · BigEarthNet | mAP: 23.13 -> 32.46 | Helped |
| use GRAFT pixel features instead of CLIPSeg | Satlas segmentation NAIP · NAIP | per-class accuracy: 26.99 -> 49.38 | Helped |
| use GRAFT pixel features instead of CLIPSeg | Satlas segmentation Sentinel-2 · Sentinel-2 | per-class accuracy: 20.02 -> 31.95 | Helped |
| add SAM refinement (ablation) | GRAFT segmentation · Satlas segmentation Sentinel-2 | per-class accuracy: 31.95 -> 32.43 | Helped |
| replace GLIP with GRAFT pixel-level model in ViperGPT | ViperGPT · RSVQA high-resolution | average accuracy: 37.69 -> 44.05 | Helped |
| sample satellite-image centers at least 112 meters apart rather than randomly (ablation) | GRAFT image-level model · NAIP-OSM internal validation | mAP: 69.3% -> 74.8% | Helped |
| replace ground-image embeddings with alt-text embeddings (ablation) | GRAFT image-level model · NAIP-OSM internal validation | mAP: 74.8% -> 65.4% maP | Hurt |
| add textual embeddings as an auxiliary loss (ablation) | GRAFT image-level model · NAIP-OSM internal validation | mAP: 74.8% -> 70% mAP | Hurt |
| initialize the satellite encoder from CLIP rather than training from scratch (ablation) | GRAFT · EuroSAT | zero-shot classification accuracy: 42.46% -> 63.76% | Helped |
| use L2 loss instead of the default alignment loss (ablation) | GRAFT image-level model · NAIP-OSM internal validation | mAP: 74.78 -> 66.80 | Hurt |
| use an average ground-image representation as the positive (ablation) | GRAFT image-level model · NAIP-OSM internal validation | mAP: 74.78 -> 71.42 | Hurt |
Surprising findings
- Aligning to ground-image CLIP embeddings outperforms aligning directly to alt-text embeddings (74.8 vs 65.4 mAP), suggesting images are a better intermediary than noisy text.
- Zero-shot GRAFT ViT-B/32 outperforms one-shot Satlas, which was pretrained on 302 million labeled examples, on BigEarthNet.
Limitations
- Cannot align dynamic objects (e.g., cargo ships, airplanes) because satellite revisit rates are low.
- Text encoder is not fine-tuned, so it may carry ground-level biases (e.g., directional concepts like 'left of').
- No text generation capability; VQA relies on external tools like ViperGPT and SAM.
Experimental setup
Datasets: Flickr ground images NAIP Sentinel-2 EuroSAT BigEarthNet SAT-4 SAT-6 OpenStreetMap SUN397 RSVQA
Benchmarks: EuroSAT BigEarthNet SAT-4 SAT-6 NAIP-OSM SatlasPretrain segmentation RSVQA
Metrics: accuracy mAP mAP@100 mAP@20 per-class accuracy
Baselines: CLIP CLIP-RSICD RemoteCLIP SeCo CACo Satlas CLIPSeg GLIP
Builds on: CLIP SAM ViperGPT SatlasPretrain
Closest papers by meaning
- Quality-Driven Curation of Remote Sensing Vision-Language Data via Learned Scoring Models (NeurIPS 2025 · similarity 0.91)
- SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing (AAAI 2024 · similarity 0.90)
- VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding (NeurIPS 2024 · similarity 0.90)
- SkySense-O: Towards Open-World Remote Sensing Interpretation with Vision-Centric Visual-Language Modeling (CVPR 2025 · similarity 0.90)
- GeoChat: Grounded Large Vision-Language Model for Remote Sensing (CVPR 2024 · similarity 0.90)
- VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysis (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.