Research map / Detection and segmentation / CNN features
RoMA: Scaling up Mamba-based Foundation Models for Remote Sensing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RoMA, a self-supervised autoregressive pretraining framework for Mamba-based remote sensing foundation models, adding adaptive rotation encoding and multi-scale token prediction, and reports scaling behavior plus gains over ViT-based RSFMs.
Problem
Self-attention's quadratic cost limits ViT-based remote sensing foundation models on high-resolution imagery, while existing Mamba RS work is limited to supervised training on small datasets; scalable self-supervised pretraining of Mamba for RS is underexplored.
Approach
Autoregressive next-token pretraining of Mamba on large unlabeled RS data, extended with an adaptive rotation encoding strategy (LBP-scored region selection, rotated crops with angle embeddings) and a multi-scale token prediction loss that aggregates tokens at higher scales.
Key results
- RoMA Mamba-B reaches 87.36% OA on AID (TR=50%) and 59.45% on UCM, above MA3E (85.86, 55.69) and SatMAE++ (85.98, 55.72).
- On 1248x1248 inputs, RoMA-Base uses 78.9% less GPU memory and runs 1.56x faster than ViT (per Figure 1 claims).
- Ablation on AID shows adaptive rotation encoding and multi-scale prediction raise OA (TR=50%) from 76.80 (baseline) to 79.16.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add adaptive rotation encoding (ARE) to the RoMA baseline (ablation) | RoMA on Mamba-B · AID | OA: 69.59 -> 71.70 (TR=20%); 76.80 -> 78.00 (TR=50%) | Helped |
| Add multi-scale prediction (MSP) alongside ARE (ablation) | RoMA on Mamba-B · AID | OA: 71.70 -> 72.69 (TR=20%); 78.00 -> 79.16 (TR=50%) | Helped |
| Use LBP rather than Wavelet or HOG as the ARE feature descriptor (ablation) | RoMA on Mamba-B · AID | OA: 71.42 / 71.94 / 71.70 (TR=20%); 77.00 / 78.32 / 78.00 (TR=50%) | Mixed |
| Use ARE patch sizes 96-64-32 rather than 96 or 96-32 (ablation) | RoMA on Mamba-B · AID | OA: 70.48 / 71.23 -> 71.70 (TR=20%); 76.82 / 77.12 -> 78.00 (TR=50%) | Helped |
| Set the ARE selection threshold to 1.0×Avg. instead of 1.5×Avg. or 0.5×Avg. (ablation) | RoMA on Mamba-B · AID | OA: 68.33 / 69.71 -> 71.70 (TR=20%); 72.39 / 74.18 -> 78.00 (TR=50%) | Helped |
| Set MSP λ to 0.1 instead of 0.01 or 1.0 (ablation) | RoMA on Mamba-B · AID | OA: 71.81 / 70.92 -> 72.69 (TR=20%); 78.23 / 77.49 -> 79.16 (TR=50%) | Helped |
| Use only the 6× additional scale in MSP rather than combinations of scales (ablation) | RoMA on Mamba-B · AID | OA: 63.17 / 68.06 / 67.81 / 72.46 -> 72.69 (TR=20%); 71.34 / 74.74 / 74.32 / 78.72 -> 79.16 (TR=50%) | Helped |
| Pretrain with RoMA rather than SatMAE++ | Mamba-B versus ViT-L · AID, TR=50% | OA: 85.98 -> 87.36 | Helped |
| Pretrain with RoMA rather than SatMAE++ | Mamba-B versus ViT-L · UCM, TR=50% | OA: 55.72 -> 59.45 | Helped |
| Pretrain with RoMA rather than SatMAE++ | Mamba-B versus ViT-L · OSCD | F1: 53.10 -> 55.63 | Helped |
| Pretrain with RoMA rather than RVSA | Mamba-B versus ViT-B+RVSA · SpaceNetv1 | mF1: 79.56 -> 79.50 | Hurt |
| Use RoMA-B rather than RingMo | UperNet backbone · iSAID | mIoU: 67.2 -> 67.4 | Helped |
Surprising findings
- Adding excessive multi-scale aggregation hurts performance; using only one larger scale with the 16x16 patch works best.
- Mamba-Large underperforms expectations on AID relative to Base, which the authors attribute to insufficient pretraining (300 epochs).
Limitations
- Evaluated mainly on optical RGB imagery; SAR and hyperspectral not covered.
- Mamba-Large scaling conclusions are inconclusive due to limited pretraining epochs; OSCD and SpaceNet scaling experiments were still ongoing.
Experimental setup
Datasets: OpticalRS-4M MillionAID AID UCM OSCD SpaceNetv1 iSAID
Benchmarks: AID UCM OSCD SpaceNetv1 iSAID
Metrics: overall accuracy F1 mF1 mIoU peak GPU memory inference speed
Baselines: MAE SatMAE++ MA3E RVSA ARM RingMo SeCo CACo GFM ScaleMAE
Builds on: iGPT ARM MAE MA3E VAR
Closest papers by meaning
- Parameter Efficient Self-Supervised Geospatial Domain Adaptation (CVPR 2024 · similarity 0.88)
- Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model (NeurIPS 2024 · similarity 0.85)
- Galileo: Learning Global & Local Features of Many Remote Sensing Modalities (ICML 2025 · similarity 0.85)
- 2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification (CVPR 2025 · similarity 0.85)
- Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion (ICLR 2025 · similarity 0.85)
- GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolution (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.