Research map / Large language models / AI and society

No One Knows the State-of-the-Art in Geospatial Foundation Models

NeurIPS 2026 · Large language models › AI and society · approach: Benchmarking and reproducibility practice · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Audits 152 geospatial foundation model (GFM) papers (2019–2025) and finds that weights, benchmarks, reported numbers, and pretraining configurations are too inconsistent to rank GFMs, then proposes six recommendations (R1–R6) and a reviewer checklist.

Problem

Published GFM papers do not share evaluation protocols, benchmarks, released weights, or pretraining controls, so it is impossible to tell which GFM is state of the art or to compare them reliably.

Approach

A structured, LLM-assisted and human-verified audit of 152 GFM papers quantifies comparability failures (weight release, benchmark fragmentation, same-protocol metric divergence, unique pretraining configurations) and turns them into concrete community expectations plus a reviewer checklist.

Key results

• Of 301 same-protocol multi-paper result tuples, 46 show a spread of at least 10 points; the largest is 56.6 points (Scale-MAE on NWPU-RESISC45 linear probe, 33.0 vs 89.6).
• 39% of the 152 GFM papers release no model weights, and another 19% ship code without model artifacts.
• Of 126 papers with an extractable pretraining dataset, 94 use a configuration no other paper uses; 35% of papers have zero overlap with the top-10 benchmarks.

Surprising findings

• The same Scale-MAE checkpoint under a nominal linear-probe protocol yields reported accuracies of 33.0 and 89.6 in different papers.
• Benchmark concentration has not improved since 2023 despite the growth of the field.

Limitations

• Metadata extraction relies on LLMs (Claude Opus 4.7, GPT 5.5 Codex) and Docling, with manual verification that may miss errors.
• Corpus excludes metadata-poor venues and many papers without weights or code, so trends may not generalize to the whole field.
• Analysis covers only public, open-source GFM literature and does not evaluate any model directly.

Experimental setup

Datasets: EuroSAT NWPU-RESISC45 AID BigEarthNet MillionAID SSL4EO-S12 fMoW GEO-Bench PANGAEA

Benchmarks: EuroSAT NWPU-RESISC45 UCMerced m-Cashew m-BigEarthNet

Metrics: accuracy mIoU F1 spread of reported metric Gini coefficient overlap with top-10 benchmarks

Builds on: GEO-Bench PANGAEA lm-evaluation-harness HELM VTAB

Closest papers by meaning

• GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data (NeurIPS 2025 · similarity 0.86)
• GeoCore-9B: Towards Geo-Aware Generative Foundation Models in Earth Observation (NeurIPS 2026 · similarity 0.86)
• REOBench: Benchmarking Robustness of Earth Observation Foundation Models (NeurIPS 2025 · similarity 0.85)
• Bridging Remote Sensors with Multisensor Geospatial Foundation Models (CVPR 2024 · similarity 0.85)
• AnySat: One Earth Observation Model for Many Resolutions, Scales, and Modalities (CVPR 2025 · similarity 0.84)
• MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models (ICML 2025 · similarity 0.84)