Research map / Vision-language and multimodal / Human and embodied modeling
Bridging Remote Sensors with Multisensor Geospatial Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes msGFM, a Swin-base masked-image-modeling foundation model pretrained on ~2M multisensor images (RGB, Sentinel-2, SAR, DSM) with cross-sensor reconstruction and MoE, evaluated on scene classification, segmentation, cloud removal and pan-sharpening.
Problem
Existing geospatial pretraining models focus on a single sensor or require paired sensor data (contrastive methods), so they cannot exploit the many unpaired sensor modalities available in practice.
Approach
Use per-sensor patch embeddings and decoders with a shared Swin encoder and SimMIM-style masked image modeling, where a masked input from one sensor may be trained to reconstruct its co-located paired sensor (cross-sensor prediction), falling back to self-reconstruction when no pair exists; add sparse Mixture-of-Experts layers.
Key results
- msGFM reaches 87.5 mAP on 10% BigEarthNet and 92.9 on 100% BigEarthNet, above reported baselines such as GFM (86.3 on 10% BEN) and SeCo (82.6 on 10% BEN).
- On SEN12MS-CR cloud removal, msGFM attains SAM 4.87 and MAE 0.026, outperforming GLF-CR (SAM 7.65) and DSen2-CR (SAM 9.47).
- Distilling features from ImageNet-22k or CLIP underperforms pretraining from scratch (e.g., Vaihingen mIoU 72.9 and 69.3 vs 74.8 from scratch).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multisensor msGFM pretraining instead of the strongest listed baseline | BigEarthNet · BigEarthNet 10% | mAP: 86.3 -> 87.5 | Helped |
| Use multisensor msGFM pretraining instead of MoCoV2 | BigEarthNet · BigEarthNet 100% | mAP: 89.3 -> 92.9 | Helped |
| Use multisensor msGFM pretraining instead of GLF-CR | SEN12MS-CR · SEN12MS-CR | SAM: 7.65 -> 4.87 | Helped |
| Pretrain from scratch rather than distilling ImageNet22K features (ablation) | msGFM · BigEarthNet 10% | mAP: 86.4 -> 87.2 | Helped |
| Pretrain from scratch rather than distilling CLIP features (ablation) | msGFM · BigEarthNet 10% | mAP: 83.8 -> 87.2 | Helped |
| Pretrain from scratch rather than reconstructing CLIP features with EVA (ablation) | msGFM · BigEarthNet 10% | mAP: 80.6 -> 87.2 | Helped |
| Add MoE to the encoder (ablation) | msGFM · BigEarthNet 1% | mAP: 78.3 -> 78.5 | No clear effect |
| Add cross-sensor reconstruction (ablation) | msGFM · BigEarthNet 1% | mAP: 78.3 -> 80.7 | Helped |
| Use 50% rather than 100% sensor crossing (ablation) | msGFM · BigEarthNet 10% | mAP: 86.8 -> 87.5 | Helped |
| Use 50% rather than 100% sensor crossing (ablation) | msGFM · SEN12MS-CR | SSIM: 0.789 -> 0.821 | Helped |
Surprising findings
- Distillation from natural-image models (ImageNet-22k, CLIP) and EVA-style CLIP-feature reconstruction underperform training from scratch on geospatial multisensor data, indicating a domain gap.
- Removing MoE yields similar performance on some datasets while other tasks drop more, showing uneven task responses.
Limitations
- Pretraining covers only spatial data; temporal information is not modeled.
- Cross-reconstruction of SAR is difficult due to noisy unprocessed SAR images and MIM's focus on low-frequency components.
Experimental setup
Datasets: GeoPile-2 SEN12MS MDAS GeoPile MillionAID BigEarthNet SEN12MS-CR Vaihingen SpaceNet 2
Benchmarks: BigEarthNet SEN12MS-CR SpaceNet Vaihingen
Metrics: mAP MAE SAM SSIM PSNR mIoU
Baselines: SeCo SatMAE MoCoV2 DINO-MC GFM ImageNet-22k pretrained Swin SAR-Opt-cGAN DSen2-CR GLF-CR CLIP distillation EVA
Builds on: SimMIM GFM Swin Transformer Sparsely-gated Mixture of Experts
Closest papers by meaning
- GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data (NeurIPS 2025 · similarity 0.88)
- SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (CVPR 2024 · similarity 0.87)
- Galileo: Learning Global & Local Features of Many Remote Sensing Modalities (ICML 2025 · similarity 0.86)
- Parameter Efficient Self-Supervised Geospatial Domain Adaptation (CVPR 2024 · similarity 0.86)
- GeoViS: Geospatially Rewarded Visual Search for Remote Sensing Visual Grounding (CVPR 2026 · similarity 0.86)
- Towards Faithful Reasoning in Remote Sensing: A Perceptually-Grounded GeoSpatial Chain-of-Thought for Vision-Language Models (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.