Research map / Vision-language and multimodal / Human and embodied modeling
SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents SkySense, a 2.06B-parameter multi-modal remote sensing foundation model pre-trained on 21.5M optical/SAR temporal sequences with multi-granularity contrastive learning and geo-context prototype learning, evaluated on 16 datasets across 7 tasks.
Problem
Existing remote sensing foundation models focus on single modalities and static imagery, neglecting temporal sequences and geo-context, which limits their use across diverse Earth observation tasks.
Approach
A factorized multi-modal spatiotemporal encoder (per-modality spatial encoders followed by a temporal fusion transformer) pre-trained with multi-granularity contrastive learning, cross-modal alignment, and unsupervised geo-context prototype learning that stores region-specific prototypes indexed by geo-location.
Key results
- SkySense reaches 70.91 mIoU on iSAID and 74.27 mAP on DIOR-R-style oriented detection benchmark (FAIR1M reported as 54.57), outperforming the prior best across all tested scenarios.
- On average SkySense beats GFM, SatLas and Scale-MAE by 2.76%, 3.67% and 3.61% respectively across evaluated tasks.
- Multi-modal fine-tuning on Dyna.-MM improves mIoU from 42.2 baseline to 48.2 with the full pre-training design (MGCL, MM, CMA, GCPL).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Multi-Granularity Contrastive Learning (MGCL) during pre-training (ablation) | SkySense · Dyna.-MM | mIoU: 42.2 -> 44.4 | Helped |
| Add multi-modal pre-training (ablation) | SkySense · Dyna.-MM | mIoU: 44.4 -> 47.0 | Helped |
| Add Cross-Modal Alignment (CMA) (ablation) | SkySense · Dyna.-MM | mIoU: +0.7% mIoU | Helped |
| Add Geo-Context Prototype Learning (GCPL) (ablation) | SkySense · Dyna.-MM | mIoU: +0.5% mIoU | Helped |
| Include SAR data in pre-training (ablation) | SkySense · iSAID | mIoU: 68.98 -> 70.91 | Helped |
| Include SAR data in pre-training (ablation) | SkySense · fMoW-S2 | Top-5 Acc: 85.69 -> 87.27 | Helped |
| Add GCP to the Dyna.-MM downstream model (ablation) | SkySense · Dyna.-MM | mIoU: +0.5% mIoU | Helped |
| Add temporal Sentinel-2 imagery instead of static multispectral data | SkySense · PASTIS-MM | OA: 73.5 -> 84.6 | Helped |
| Add Sentinel-1 temporal data to Sentinel-2 temporal data | SkySense · PASTIS-MM | OA: 84.6 -> 84.8 | No clear effect |
| Add Sentinel-1 data when cloud ratio exceeds 50% | SkySense · PASTIS-MM | OA: 13% | Helped |
| Add Sentinel-2 imagery to Sentinel-1 imagery | SkySense · BEN-MM | mAP: 86.25 -> 92.21 | Helped |
Surprising findings
- Integrating Sentinel-1 SAR data gave no substantial gain on PASTIS-MM, attributed to cloud-free imagery, though SAR helped considerably when cloud ratio exceeded 50%.
Limitations
- Does not incorporate the language modality, as noted by the authors' future work.
- Several baseline models' code and weights were not released, so some numbers are taken from papers.
Experimental setup
Datasets: WorldView-3/4 HSROI Sentinel-2 time series Sentinel-1 SAR time series Dyna.-Pla. iSAID Potsdam Dyna.-S2 DIOR DIOR-R FAIR1M LEVIR-CD OSCD
Benchmarks: iSAID Potsdam Dyna.-Pla. Dyna.-S2 DIOR DIOR-R FAIR1M LEVIR-CD OSCD AID RESISC-45 BEN-S2
Metrics: mIoU mAP mAP50 F1 OA Top-1 accuracy Top-5 accuracy SCS
Baselines: GASSL SeCo SatMAE RingMo RVSA CMID CACo SatLas GFM Scale-MAE
Builds on: DINO MoCo-v2 Swin Transformer Sinkhorn-Knopp clustering SwAV-style prototypes
Closest papers by meaning
- SkySense-O: Towards Open-World Remote Sensing Interpretation with Vision-Centric Visual-Language Modeling (CVPR 2025 · similarity 0.90)
- RobSense: A Robust Multi-modal Foundation Model for Remote Sensing with Static, Temporal, and Incomplete Data Adaptability (CVPR 2025 · similarity 0.90)
- Galileo: Learning Global & Local Features of Many Remote Sensing Modalities (ICML 2025 · similarity 0.89)
- SkyMoE: A Vision-Language Foundation Model for Enhancing Geospatial Interpretation with Mixture of Experts (AAAI 2026 · similarity 0.88)
- GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data (NeurIPS 2025 · similarity 0.88)
- SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.