Research map / Detection and segmentation / Open-set and few-shot detection
FoMo: Multi-Modal, Multi-Scale and Multi-Task Remote Sensing Foundation Models for Forest Monitoring
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FoMo-Bench, a 15-dataset forest monitoring benchmark for remote sensing foundation models, the TalloS tree-genus dataset, and FoMo-Net, a MAE-based pretraining framework with random spectral-band sampling over a ViT encoder.
Problem
Evaluating and building remote sensing foundation models that handle variable modalities, spectral bands, ground sampling distances and tasks for forest monitoring, which lacks a unified global benchmark.
Approach
Treat each spectral band of each sensor as a separate modality, sample a random variable subset of bands and datasets per batch, tokenize each band independently with a shared linear projection plus learned spectral and positional embeddings, and pretrain a ViT masked autoencoder with gradient accumulation across batches.
Key results
- On TalloS, FoMo-Net1 reaches 18.50 micro F1 versus 27.31 for supervised ResNet50, with ViT at 15.23.
- On RapidAI4EO, FoMo-Net1 gets 54.69 micro F1 versus 64.07 for ConvNext and 51.04 for ViT, exceeding the original ViT.
- FoMo-Net1 (single shared projection) outperforms per-band projections FoMo-Net_m across most classification and detection tables.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use one shared linear projection rather than separate projections for each spectral band (ablation) | FoMo-Net · BigEarthNet | F1 micro: 66.11 ($\pm0.07$) -> 73.42 ($\pm0.10$) | Helped |
| Use one shared linear projection rather than separate projections for each spectral band (ablation) | FoMo-Net · Sen12MS | F1 micro: 38.07 ($\pm0.65$) -> 42.87 ($\pm1.69$) | Helped |
| Use one shared linear projection rather than separate projections for each spectral band (ablation) | FoMo-Net · TalloS | F1 micro: 14.25 ($\pm2.81$) -> 18.50 ($\pm 2.33$) | Helped |
| Use one shared linear projection rather than separate projections for each spectral band (ablation) | FoMo-Net · RapidAI4EO | F1 micro: 49.12 ($\pm5.59$) -> 54.69 ($\pm0.06$) | Helped |
| Use one shared linear projection rather than separate projections for each spectral band (ablation) | FoMo-Net · TreeSatAI | F1 micro: 27.96 ($\pm1.02$) -> 45.02 ($\pm 0.51$) | Helped |
| Use one shared linear projection rather than separate projections for each spectral band (ablation) | FoMo-Net · Waititu | mIoU: 55.66 ($\pm0.0$) -> 59.10 ($\pm 2.51$) | Helped |
| Use one shared linear projection rather than separate projections for each spectral band (ablation) | FoMo-Net · FLAIR #1 | mIoU: 43.08 ($\pm0.39$) -> 49.43 ($\pm0.06$) | Helped |
| Use one shared linear projection rather than separate projections for each spectral band (ablation) | FoMo-Net · FLAIR #2 | mIoU: 40.80 ($\pm0.16$) -> 47.81 ($\pm0.08$) | Helped |
| Use one shared linear projection rather than separate projections for each spectral band (ablation) | FoMo-Net · Spekboom | mIoU: 70.97 ($\pm1.15$) -> 91.56 ($\pm0.11$) | Helped |
| Use one shared linear projection rather than separate projections for each spectral band (ablation) | FoMo-Net · FiveBillionPixels | mIoU: 50.82 ($\pm0.34$) -> 57.18 ($\pm0.24$) | Helped |
| Use one shared linear projection rather than separate projections for each spectral band (ablation) | FoMo-Net · ForestNet | mIoU: 90.92 ($\pm0.0$) -> 90.89 ($\pm0.04$) | No clear effect |
| Use one shared linear projection rather than separate projections for each spectral band (ablation) | FoMo-Net · Woody | mIoU: 72.93 ($\pm1.27$) -> 81.32 ($\pm0.06$) | Helped |
Surprising findings
- A single shared linear projection for all spectral bands (FoMo-Net1) outperforms dedicated per-band projections (FoMo-Net_m), contrary to intuition.
- Traditional supervised CNNs consistently rank among the top performers, with FoMo-Net generally trailing specialized supervised models.
Limitations
- Token count scales with number of spectral bands, causing memory and quadratic attention costs.
- FoMo-Net still underperforms task-specific supervised baselines on most tasks.
- Point-cloud data are excluded from pretraining and the model is not applied to non-grid data.
Experimental setup
Datasets: BigEarthNet-MM Sen12MS RapidAI4EO ForestNet FiveBillionPixels TreeSatAI TalloS NeonTree Woody ReforesTree Spekboom Waititu
Benchmarks: FoMo-Bench BigEarthNet-MM Sen12MS RapidAI4EO TreeSatAI TalloS Waititu FLAIR #1 FLAIR #2 Spekboom FiveBillionPixels ForestNet
Metrics: F1 micro F1 macro mIoU mAP50 mAP50:95
Baselines: ResNet50 ViT ConvNext PolyNet UNet DeepLabv3 UPerNet Faster R-CNN RetinaNet YOLOS DINOv2
Builds on: masked autoencoder vision transformer PolyNet Geo-Bench
Closest papers by meaning
- Open-Canopy: Towards Very High Resolution Forest Monitoring (CVPR 2025 · similarity 0.86)
- OAM-TCD: A globally diverse dataset of high-resolution tree cover maps (NeurIPS 2024 · similarity 0.86)
- Can Large Multimodal Models Understand Agricultural Scenes? Benchmarking with AgroMind (NeurIPS 2025 · similarity 0.85)
- SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing (AAAI 2024 · similarity 0.85)
- Multi-modal Learning for Geospatial Vegetation Forecasting (CVPR 2024 · similarity 0.85)
- Bringing SAM to new heights: leveraging elevation data for tree crown segmentation from drone imagery (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.