Research map / Vision-language and multimodal / Human and embodied modeling
Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SatMAE++, a multi-scale masked autoencoder pre-training for optical and multi-spectral satellite imagery that reconstructs images at multiple resolutions via convolutional upsampling blocks, reaching SOTA on six remote sensing benchmarks.
Problem
Existing satellite-image masked autoencoder pre-training (e.g., SatMAE) ignores multi-scale (GSD) information and does not work across both RGB and multi-spectral data, while GSD-based positional encodings (ScaleMAE) only apply to RGB.
Approach
Keep the standard sinusoidal positional encoding of SatMAE and add extensible transpose-convolution upsampling blocks that reconstruct the image at 2x and 4x resolutions with L1 losses, so the encoder learns multi-scale features for both optical and multi-spectral data.
Key results
- SatMAE++ reaches 78.14% top-1 on fMoW-RGB, above ScaleMAE's 77.9%, and 63.23% on fMoW-Sentinel vs. SatMAE 61.61% (reproduced).
- On BigEarthNet multi-label classification, SatMAE++ achieves 85.11 mAP vs. SatMAE 82.13 (about +2.5-3 points).
- SatMAE++ reaches 97.48% on RESISC-45 (vs. ScaleMAE 95.7%) and 97.62% on UC-Merced (vs. SatMAE 94.05%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multi-scale pre-training | SatMAE with ViT-Large · fMoW-RGB validation | Top1 Acc.: 73.87 -> 78.14 | Helped |
| Add multi-scale pre-training | SatMAE with ViT-Large · fMoW-Sentinel validation | Top1 Acc.: 61.48 -> 63.23 | Helped |
| Add multi-scale pre-training | SatMAE with ViT-Base · fMoW-Sentinel validation | Top1 Acc.: 60.01 -> 62.69 | Helped |
| Use two pre-training scales instead of one (ablation) | SatMAE++ with ViT-Large · fMoW-Sentinel validation | Top1 Acc.: 61.61 -> 62.86 | Helped |
| Use three pre-training scales instead of two (ablation) | SatMAE++ with ViT-Large · fMoW-Sentinel validation | Top1 Acc.: 62.86 -> 63.23 | Helped |
| Add multi-scale pre-training | SatMAE with ViT-Large · EuroSAT | Top1 Acc.: 95.74 -> 99.04 | Helped |
| Add multi-scale pre-training | SatMAE with ViT-Large · RESISC-45 | Top1 Acc.: 94.8 -> 97.48 | Helped |
| Add multi-scale pre-training | SatMAE with ViT-Large · UC-Merced | Top1 Acc.: 94.05 -> 97.62 | Helped |
| Add multi-scale pre-training | SatMAE with ViT-Large · BigEarthNet | mAP: 82.13 -> 85.11 | Helped |
Surprising findings
- Multi-scale pre-training makes the model converge faster during finetuning on fMoW-Sentinel, reaching its best score at epoch 12 versus epoch 20 for single-scale pre-training.
- The simpler standard positional encoding with multi-scale reconstruction outperforms ScaleMAE's GSD-based positional encoding and Laplacian decoder on fMoW-RGB without those components.
Limitations
- Struggles in some cases to reconstruct fine-grained structural details at higher scales.
- Gains from adding a third scale are small (0.37%), suggesting diminishing returns from more scales.
- Only evaluated on classification tasks; dense prediction is left to future work.
Experimental setup
Datasets: fMoW-RGB fMoW-Sentinel EuroSAT RESISC-45 UC-Merced BigEarthNet
Benchmarks: fMoW-RGB fMoW-Sentinel EuroSAT RESISC-45 UC-Merced BigEarthNet
Metrics: top-1 accuracy mean average precision
Baselines: SatMAE ScaleMAE ConvMAE MAE GASSL SeCo MoCo-V3 MoCo-V2
Builds on: SatMAE MAE ScaleMAE ConvMAE
Closest papers by meaning
- S2MAE: A Spatial-Spectral Pretraining Foundation Model for Spectral Remote Sensing Data (CVPR 2024 · similarity 0.87)
- SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (CVPR 2024 · similarity 0.86)
- Parameter Efficient Self-Supervised Geospatial Domain Adaptation (CVPR 2024 · similarity 0.85)
- MMM-RS: A Multi-modal, Multi-GSD, Multi-scene Remote Sensing Dataset and Benchmark for Text-to-Image Generation (NeurIPS 2024 · similarity 0.85)
- Galileo: Learning Global & Local Features of Many Remote Sensing Modalities (ICML 2025 · similarity 0.85)
- Data Augmentation Approaches for Satellite Imagery (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.