Research map / Vision-language and multimodal / Human and embodied modeling
S2MAE: A Spatial-Spectral Pretraining Foundation Model for Spectral Remote Sensing Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S2MAE, a 3D-masked-autoencoder ViT pretrained progressively on fMoW-S2 and BigEarthNet Sentinel-2 imagery with a 90% mask ratio, evaluated on EuroSAT, BigEarthNet, and OSCD change detection.
Problem
Standard pretrained vision models target natural RGB images and transfer poorly to multispectral/hyperspectral remote sensing data, whose bands have spectral sequencing and spatial alignment; existing group-based masking (SatMAE) limits spectral interaction and band flexibility.
Approach
Extends MAE to spectral imagery with 3D (spatial-spectral) random patch masking over cube tokens, separable spatial and spectral positional embeddings, and progressive continued pretraining across datasets of different image sizes and regions.
Key results
- On EuroSAT, S2MAE* (progressive pretraining) reaches 99.19% accuracy versus 99.09% for SatMAE.
- On BigEarthNet multi-label classification with 10% of training data, S2MAE* achieves 87.41 mAP versus 84.93 for SatMAE.
- On OSCD change detection, S2MAE* attains the highest F1 of 54.26, surpassing the second-best model by 0.52 points; ViT-H with S2MAE* reaches 90.72 mAP on BigEarthNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Progressively pretrain on fMoW-S2 and BigEarthNet instead of fMoW-S2 alone (ablation) | S2MAE · BigEarthNet | mAP: 85.59 -> 87.41 | Helped |
| Progressively pretrain on fMoW-S2 and BigEarthNet instead of fMoW-S2 alone | S2MAE · EuroSAT validation | accuracy (%): 99.16 -> 99.19 | Helped |
| Progressively pretrain on fMoW-S2 and BigEarthNet instead of fMoW-S2 alone | S2MAE · OSCD | F1: 53.28 -> 54.26 | Helped |
| Use a 90% rather than 25%, 50%, or 75% masking ratio (ablation) | S2MAE · BigEarthNet | mAP: 82.81; 84.32; 84.94 -> 85.59 | Helped |
| Use patch size 8 instead of 16 (ablation) | S2MAE · BigEarthNet | mAP: 78.42 -> 85.59 | Helped |
| Add normalized reconstruction targets instead of raw data (ablation) | S2MAE · BigEarthNet | mAP: 84.86 -> 85.59 | Helped |
| Use standardized rather than normalized reconstruction targets (ablation) | S2MAE · BigEarthNet | mAP: 85.59 -> 85.52 | No clear effect |
| Increase decoder depth from 2 to 4 blocks (ablation) | S2MAE · BigEarthNet | mAP: 84.91 -> 85.59 | Helped |
| Increase decoder depth from 4 to 8 blocks (ablation) | S2MAE · BigEarthNet | mAP: 85.59 -> 85.47 | Hurt |
| Use vanilla sin-cos instead of learnable spatial-spectral positional embeddings (ablation) | S2MAE · BigEarthNet | mAP: 85.59 -> 85.57 | No clear effect |
| Use spatial-spectral sin-cos instead of learnable spatial-spectral positional embeddings (ablation) | S2MAE · BigEarthNet | mAP: 85.59 -> 85.62 | No clear effect |
| Pretrain on fMoW-S2 rather than BigEarthNet only (ablation) | S2MAE · BigEarthNet | mAP: 83.11 -> 85.59 | Helped |
Surprising findings
- Raw (unnormalized) reconstruction targets significantly underperform normalized or standardized targets.
- S2MAE has low precision on OSCD change detection despite high recall and F1, attributed to data imbalance and very few training images.
Limitations
- Complexity remains high for downstream tasks despite cheaper 90% masking during pretraining.
- Masking strategies for longer spectral sequences are not explored.
- Low precision on OSCD due to very small training set (14 images) and overfitting risk for ViT.
Experimental setup
Datasets: fMoW-Sentinel (fMoW-S2) BigEarthNet EuroSAT OSCD
Benchmarks: EuroSAT BigEarthNet OSCD
Metrics: top-1 accuracy mAP precision recall F1
Baselines: ResNet50 SeCo ViT from scratch ViT ImageNet-22k SatMAE
Builds on: MAE MAE-ST SatMAE ViT
Closest papers by meaning
- LESSViT: Robust Hyperspectral Representation Learning under Spectral Configuration Shift (NeurIPS 2026 · similarity 0.88)
- Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery (CVPR 2024 · similarity 0.87)
- Accounting for Spatial Variability with the Histogram of Oriented Gradients Based Masking Improves Performance of Masked Autoencoder over Hyperspectral Satellite Imagery (Student Abstract) (AAAI 2025 · similarity 0.86)
- SSDiff: Spatial-spectral Integrated Diffusion Model for Remote Sensing Pansharpening (NeurIPS 2024 · similarity 0.86)
- HetSSNet: Spatial-Spectral Heterogeneous Graph Learning Network for Panchromatic and Multispectral Images Fusion (ICML 2025 · similarity 0.85)
- EigenSR: Eigenimage-Bridged Pre-Trained RGB Learners for Single Hyperspectral Image Super-Resolution (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.