Research map / Vision-language and multimodal / Cross-modal learning
RobSense: A Robust Multi-modal Foundation Model for Remote Sensing with Static, Temporal, and Incomplete Data Adaptability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RobSense, a ViT-based MS-SAR remote sensing foundation model pre-trained with temporal distribution alignment (KL divergence) plus MAE and latent reconstructors that recover representations from incomplete band/time inputs.
Problem
Remote sensing foundation models support only limited input types (static vs temporal, uni-modal vs multi-modal) and degrade on incomplete data with missing spectral bands or irregular temporal sequences.
Approach
Replaces contrastive MS-SAR alignment with a temporal distribution alignment loss that matches per-timestep Gaussian distributions of the two modalities via KL divergence, then trains separate uni-modal latent reconstructors on incomplete inputs (dropped bands and time points) so the model recovers rich representations from sparse ones.
Key results
- On complete data, RobSense reaches 46.36 mIoU in multi-modal temporal segmentation, outperforming SpectralGPT by 1.47% and Croma by 5.34%.
- On BigEarthNet multi-modal static classification, RobSense achieves 88.17 mAP on complete data and 84.87 mAP under random missing rate.
- On OSCD change detection, RobSense achieves F1 58.22 on complete data, outperforming SatMAE++ and SpectralGPT by roughly 2-4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RobSense with temporal multi-modal input | RobSense · SatlasPretrain segmentation, complete dataset | mIoU: 44.94 -> 46.36 | Helped |
| use RobSense with temporal multi-modal input | RobSense · SatlasPretrain segmentation, Rand. missing | mIoU: 42.96 | Helped |
| use RobSense with multi-modal input | RobSense · BigEarthNet classification, complete dataset | mAP: 86.28 -> 88.17 | Helped |
| use RobSense with multi-modal input | RobSense · BigEarthNet classification, Rand. missing | mAP: 82.33 -> 84.87 | Helped |
| use RobSense | RobSense · OSCD change detection, complete MS dataset | F1: 56.28 -> 58.22 | Helped |
| use RobSense | RobSense · OSCD change detection, Rand. missing MS dataset | F1: 53.20 -> 55.47 | Helped |
| add TDA to AE pre-training (ablation) | RobSense · BigEarthNet classification, static MS-SAR | mAP: 81.11 -> 84.19 | Helped |
| add contrastive loss to AE pre-training (ablation) | RobSense · BigEarthNet classification, static MS-SAR | mAP: 81.11 -> 81.53 | Helped |
| add latent reconstructors (ablation) | RobSense · BigEarthNet classification, incomplete static MS-SAR | mAP: 78.67 -> 80.70 | Helped |
Surprising findings
- Croma's performance declines when MS-SAR data is combined compared to MS alone on BigEarthNet, attributed to patch-level misalignment.
Limitations
- Requires higher computational cost for pre-training due to two pre-training phases.
- Evaluated only on three downstream tasks and MS-SAR modalities; no language extension.
- Incomplete datasets are simulated by random removal since real incomplete dataset is not public.
Experimental setup
Datasets: SatlasPretrain BigEarthNet OSCD
Benchmarks: SatlasPretrain segmentation BigEarthNet OSCD
Metrics: mIoU mAP F1
Baselines: SatlasNet SpectralGPT SatMAE++ Croma ConvNeXtV2
Builds on: masked autoencoder SatMAE Croma UperNet
Closest papers by meaning
- SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (CVPR 2024 · similarity 0.90)
- Galileo: Learning Global & Local Features of Many Remote Sensing Modalities (ICML 2025 · similarity 0.85)
- Parameter Efficient Self-Supervised Geospatial Domain Adaptation (CVPR 2024 · similarity 0.85)
- LDS2AE: Local Diffusion Shared-Specific Autoencoder for Multimodal Remote Sensing Image Classification with Arbitrary Missing Modalities (AAAI 2024 · similarity 0.84)
- SARDet-100K: Towards Open-Source Benchmark and ToolKit for Large-Scale SAR Object Detection (NeurIPS 2024 · similarity 0.84)
- Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.