Research map / Vision-language and multimodal / Human and embodied modeling
REOBench: Benchmarking Robustness of Earth Observation Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces REOBench, a benchmark evaluating robustness of Earth observation foundation models (MIM, contrastive, and MLLM based) on six RS tasks under twelve synthetic corruptions at five severity levels.
Problem
Earth observation foundation models are rarely evaluated for robustness to realistic remote sensing perturbations such as clouds, haze, blur, noise, and geometric shifts.
Approach
Builds a corrupted version of AID, ISPRS Potsdam, DIOR, and VRSBench with 12 corruption types across five severities, and measures relative task performance drop across MIM-, CL-, and LLM-based EO foundation models.
Key results
- Vision-language models (CL- and LLM-based) show smaller drops than MIM-based models; MIM scene classification drops exceed 25% on average, while VLMs mostly stay below 10%.
- Motion blur causes the largest drop, with MIM models losing around 60% of performance in some cases.
- GeoRSCLIP ViT-L achieves best clean and corrupted scene classification (91.80% avg), while VHM has the lowest relative drop among LLM-based models (4.81%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a vision-language model instead of a MIM-based model | Earth observation foundation models · AID scene classification | R_TP | Helped |
| Use a vision-language model instead of a MIM-based model | Earth observation foundation models · ISPRS Potsdam semantic segmentation | R_TP | Helped |
| Use a larger ViT-L backbone instead of ViT-B | Foundation models · Scene classification | R_TP | Helped |
| Use a larger ViT-L backbone instead of ViT-B | Foundation models · Semantic segmentation and object detection | R_TP | Hurt |
| Fine-tune GeoChat on the VRSBench training set | GeoChat · VRSBench-Cap image captioning | Avg CLAIR: 39.59 -> 68.15 | Helped |
| Fine-tune GeoChat on the VRSBench training set | GeoChat · VRSBench-Cap image captioning | R_TP: 4.35 -> 4.36 | No clear effect |
| Fine-tune GeoChat on the VRSBench training set | GeoChat · VRSBench-VQA | Avg accuracy: 55.06 -> 74.65 | Helped |
| Fine-tune GeoChat on the VRSBench training set | GeoChat · VRSBench-VQA | R_TP | Helped |
| Fine-tune GeoChat on the VRSBench training set | GeoChat · VRSBench-Ref visual grounding | Avg accuracy at IoU 0.5: 16.72 -> 52.36 | Helped |
| Apply all three corruptions simultaneously instead of clean input | RemoteCLIP · DIOR-R object detection | mAP: 60.40 -> 40.58 | Hurt |
| Combine Brightness and Compression | RemoteCLIP · DIOR-R object detection | mAP | Hurt |
| Expose models to image corruptions | Earth observation foundation models · Multispectral scene classification | task performance | Hurt |
Surprising findings
- Larger ViT-L backbones are more robust for scene classification but suffer larger drops than ViT-B on segmentation and detection.
- Compound corruptions produce synergistic effects that exceed the sum of individual drops.
Limitations
- Evaluation limited to high-resolution optical RGB imagery; multispectral, hyperspectral, and SAR only in preliminary experiments.
- Task coverage not exhaustive (no change detection, region captioning, or counting).
- Source datasets may not reflect global geographic, temporal, and sensor diversity.
Experimental setup
Datasets: AID ISPRS Potsdam DIOR-R VRSBench fMoW-Sentinel2 BigEarthNet
Benchmarks: AID ISPRS Potsdam DIOR-R VRSBench-Cap VRSBench-VQA VRSBench-Ref fMoW-Sentinel2 BigEarthNet
Metrics: accuracy mIoU mAP CLAIR relative task performance drop (R_TP) FID
Baselines: SatMAE ScaleMAE RVSA SatMAE++ RemoteCLIP GeoRSCLIP GeoChat LHRS-Bot RS-LLaVA SkySenseGPT VHM Falcon
Builds on: MAE CLIP LLaVA RVSA
Closest papers by meaning
- CHOICE: Benchmarking the Remote Sensing Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- GeoCore-9B: Towards Geo-Aware Generative Foundation Models in Earth Observation (NeurIPS 2026 · similarity 0.88)
- See What We Cannot See: A Geo-guided Reasoning Benchmark for Object Counting under Adverse Earth Observation Conditions (CVPR 2026 · similarity 0.87)
- Towards Faithful Reasoning in Remote Sensing: A Perceptually-Grounded GeoSpatial Chain-of-Thought for Vision-Language Models (ICLR 2026 · similarity 0.87)
- Mars-Bench: A Benchmark for Evaluating Foundation Models for Mars Science Tasks (NeurIPS 2025 · similarity 0.86)
- VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.