Research map / Vision-language and multimodal / CLIP and zero-shot
SkySense-O: Towards Open-World Remote Sensing Interpretation with Vision-Centric Visual-Language Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds the Sky-SA open-text pixel-level remote sensing dataset and SkySense-O, a vision-centric CLIP-style model whose pretraining keeps visual self-supervision and whose fine-tuning uses a GPT-4-built visual-relevance graph to sample hard negatives, achieving zero-shot SOTA on 14 RS datasets.
Problem
Open-world remote sensing interpretation is limited by few semantic categories in pixel-level RS datasets and by language-only alignment that confuses spatially and contextually related but visually different RS categories.
Approach
Introduce a vision-centric principle into RS vision-language modeling: keep visual contrastive and masked-image-modeling losses during image-text pretraining, and build a GPT-4-derived visual-relevance knowledge graph over category texts to sample hard negatives and weight a vision-centric regularization loss during pixel-level fine-tuning.
Key results
- Zero-shot object segmentation: SkySense-O reaches 43.92 mIoU on iSAID and outperforms prior zero-shot SOTAs by about 14.27% on average across object segmentation datasets.
- Zero-shot scene classification: 83.28% on RESISC-45 and 82.93% on AID top-1, up 9.38 and 6.66 points over prior SOTA.
- Ablation on iSAID: the vision-centric regularization loss raises mIoU from 41.69 to 43.92; the vision-centric graph sampler adds 8.52 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Sky-SA dataset for fine-tuning (ablation) | CLIP fine-tuned on COCO-Stuff · iSAID | mIoU: 18.40 -> 27.76 | Helped |
| Add vision-centric graph sampler (ablation) | CLIP fine-tuned on Sky-SA · iSAID | mIoU: 27.76 -> 36.28 | Helped |
| Use pretrained remote-sensing foundation model (ablation) | Sky-SA + vision-centric graph model · iSAID | mIoU: 36.28 -> 41.69 | Helped |
| Add vision-centric regularization loss (ablation) | SkySense-O · iSAID | mIoU: 41.69 -> 43.92 | Helped |
| Add pixel-level fine-tuning by Sky-SA (ablation) | SkySense-O visual encoder in the VQA model · SIRI-WHU | Acc.: 72.45 -> 74.79 | Helped |
| Add pixel-level fine-tuning by Sky-SA (ablation) | SkySense-O visual encoder in the VQA model · AID-VQA | Acc.: 92.95 -> 94.10 | Helped |
| Add pixel-level fine-tuning by Sky-SA (ablation) | SkySense-O visual encoder in the VQA model · RSVQA-HR | Avg. Acc.: 75.39 -> 78.09 | Helped |
| Use SkySense-O instead of SegEarth-OV | Zero-shot RS semantic segmentation · iSAID | mIoU (%): 21.70 -> 43.92 | Helped |
| Use SkySense-O instead of the strongest listed baseline | Zero-shot RS semantic segmentation · SIOR | mIoU (%): 37.47 -> 30.89 | Hurt |
| Use SkySense-O instead of the strongest listed baseline | Zero-shot RS semantic segmentation · FloodNet | mIoU (%): 44.55 -> 42.04 | Hurt |
Surprising findings
- Sky-SA-trained models generalize poorly to other datasets' test sets when trained on existing datasets, while Sky-SA gives the best average zero-shot transfer (over 13.43% mIoU above the second-best dataset).
Limitations
- Sky-SA is not stated as publicly released, limiting reproducibility of the main dataset.
- Vision-centric graph relies on GPT-4 judgments of visual relevance, which may introduce bias.
- Main gains on some land-cover datasets are modest (e.g., +0.53 on OEM over CAT-SEG-like baselines).
Experimental setup
Datasets: SkyScript RS5M OSM Sky-SA COCO-Stuff CC-558K ChatEarthNet SkySenseGPT instruction data iSAID FAST SIOR SOTA
Benchmarks: iSAID FAST SIOR SOTA ISPRS Potsdam Vaihingen LoveDA OpenEarthMap FloodNet AID RESISC-45 SIRI-WHU-VQA
Metrics: mIoU Top-1 accuracy VQA accuracy
Baselines: SegEarth-OV CAT-SEG GeoRSCLIP VHM SkyScript RemoteCLIP SegGPT DINOv SAN GeoChat LHRS-Bot EarthGPT
Builds on: SkySense CLIP CAT-Seg MaskCLIP LLaVA-1.5 DetailCLIP
Closest papers by meaning
- SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing (AAAI 2024 · similarity 0.93)
- VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding (NeurIPS 2024 · similarity 0.91)
- SkyMoE: A Vision-Language Foundation Model for Enhancing Geospatial Interpretation with Mixture of Experts (AAAI 2026 · similarity 0.91)
- SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (CVPR 2024 · similarity 0.90)
- Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment (ICLR 2024 · similarity 0.90)
- Landsat30-AU: A Vision-Language Dataset for Australian Landsat Imagery (AAAI 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.