Research map / Vision-language and multimodal / CLIP and zero-shot
SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds SkyScript, a 2.6M image-text remote sensing dataset by linking GEE imagery to OpenStreetMap tags via geo-coordinates, and continually pre-trains CLIP into SkyCLIP for zero-shot remote sensing tasks.
Problem
Vision-language models for remote sensing lack large, semantically diverse image-text data, since remote sensing images cannot be crawled from the web at scale and human annotation is costly.
Approach
Pair open unlabeled GEE remote sensing images with OpenStreetMap tags via geo-coordinates, filter tags with CLIP-embedding classifiers for visual groundability and max GSD, assemble captions from tags, and filter pairs by CLIP similarity, then continually pre-train CLIP on the result.
Key results
- SkyCLIP-50 (ViT-L-14) reaches 59.93% average zero-shot scene classification accuracy over seven benchmarks, a 6.17-point gain over CLIP-original (53.76%).
- SkyCLIP-50 outperforms RemoteCLIP by 9.98% average accuracy and human-curated captions by 4.52%.
- SkyCLIP outperforms the supervised cross-modal models AMFMN, LW-MCR, and GaLR on UCM-Captions by more than 10% (text-to-image) and 25% (image-to-text).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Continual pre-train CLIP on SkyScript rather than use the original CLIP | CLIP/ViT-L-14 · Seven scene-classification benchmarks | average top-1 accuracy (%): 53.76 -> 59.93 | Helped |
| Continual pre-train on SkyScript rather than LAION-RS | CLIP/ViT-L-14 · Seven scene-classification benchmarks | average top-1 accuracy (%): 57.87 -> 59.93 | Helped |
| Train on SkyScript rather than human-curated captions | CLIP/ViT-L-14 · Seven scene-classification benchmarks | average top-1 accuracy (%): 55.41 -> 59.93 | Helped |
| Continual pre-train on SkyScript rather than RemoteCLIP training | CLIP/ViT-L-14 · Seven scene-classification benchmarks | average top-1 accuracy (%): 49.95 -> 59.93 | Helped |
| Increase the retained SkyScript fraction from 20% to 50% (ablation) | SkyCLIP · Seven scene-classification benchmarks | average top-1 accuracy (%): 59.81 -> 59.93 | Helped |
| Use all SkyScript pairs instead of filtering to the top 20% (ablation) | SkyCLIP · Seven scene-classification benchmarks | average top-1 accuracy (%): 59.81 -> 58.65 | Hurt |
| Use CLIP-laion-RS rather than original CLIP to score pairs for filtering (ablation) | SkyCLIP-30 · Seven scene-classification benchmarks | average top-1 accuracy (%): 59.91 -> 60.69 | Helped |
| Freeze more layers in the image encoder or text encoder (ablation) | SkyCLIP · Scene classification benchmarks | accuracy | Hurt |
| Freeze the image encoder rather than the text encoder (ablation) | SkyCLIP · Scene classification benchmarks | accuracy | Hurt |
| Continual pre-train on SkyScript rather than use original CLIP | CLIP/ViT-B-32 · Roof shape | top-1 accuracy (%): 31.50 -> 26.00 | Hurt |
| Continual pre-train on SkyScript rather than use original CLIP | CLIP · Zero-shot cross-modal retrieval on RSICD and RSITMD | mean recall (%): +2.57% to +6.48% | Helped |
| Exclude 10 classes from SkyScript training (ablation) | SkyCLIP · Unseen classes in Million-AID | accuracy (%): 93.6% -> 96.1% | Helped |
Surprising findings
- Using the full unfiltered 5.2M pairs yields lower average accuracy (58.65%) than the top-20% or top-50% filtered subsets, showing very-low-similarity pairs hurt.
- RemoteCLIP scores 34.40% on SkyScript classification with ViT-L-14 despite strong results on some benchmarks, indicating uneven transfer.
Limitations
- Geographic bias: high-resolution imagery concentrated in the U.S. and Europe, and OSM annotations less complete in developing countries.
- Captions are assembled by simple rule-based concatenation of tags, lacking natural language structure and possibly making models bag-of-words-like.
- Tag classifiers have modest accuracy (GSD classifier cross-validation accuracy 0.53).
Experimental setup
Datasets: SkyScript LAION-RS AID EuroSAT fMoW Million-AID PatternNet RESISC45 RSI-CB256 RSICD RSITMD UCM-Captions
Benchmarks: AID EuroSAT fMoW Million-AID PatternNet RESISC45 RSI-CB256 SkyScript-classification UCM-Captions RSICD RSITMD SkyScript-retrieval
Metrics: top-1 accuracy mean recall recall@1 recall@5 recall@10
Baselines: CLIP-original RemoteCLIP CLIP-laion-RS Human-curated captions AMFMN LW-MCR GaLR
Builds on: CLIP RemoteCLIP LAION-2B
Closest papers by meaning
- SkySense-O: Towards Open-World Remote Sensing Interpretation with Vision-Centric Visual-Language Modeling (CVPR 2025 · similarity 0.93)
- VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding (NeurIPS 2024 · similarity 0.91)
- Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment (ICLR 2024 · similarity 0.90)
- Quality-Driven Curation of Remote Sensing Vision-Language Data via Learned Scoring Models (NeurIPS 2025 · similarity 0.90)
- Landsat30-AU: A Vision-Language Dataset for Australian Landsat Imagery (AAAI 2026 · similarity 0.90)
- VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysis (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.