Research map / Time series / Foundation models for time series
Multi-modal Learning for Geospatial Vegetation Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the GreenEarthNet dataset and Contextformer, a multi-modal transformer that forecasts Sentinel-2 NDVI vegetation greenness from satellite history and meteorology, outperforming video prediction and EarthNet2021 baselines.
Problem
Fine-resolution vegetation forecasting has been studied mainly for perceptual image quality rather than vegetation dynamics, and there is no dataset designed for within-year vegetation dynamics prediction at high resolution.
Approach
A PVT-v2 vision encoder for spatial context, a patchwise temporal transformer over local 4x4 px patches with meteorological embeddings, and a delta-prediction decoder that predicts deviations from the last cloud-free NDVI, paired with a new dataset with a learned cloud mask and vegetation-focused evaluation.
Key results
- Contextformer reaches R2 0.62, RMSE 0.14 and NSE 0.09 on OOD-t, the best among all compared models, with 66.8% outperformance over climatology (6.1M params).
- Earthformer, SGED-ConvLSTM and ConvLSTM trained on EarthNet2021 fail to consistently beat climatology (R2 0.49 to 0.53).
- Shuffling spatial inputs reduces Contextformer R2 from 0.62 to 0.55, indicating the model uses spatial interactions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Contextformer instead of the strongest reported baseline, PredRNN | GreenEarthNet OOD-t vegetation forecasting · GreenEarthNet OOD-t | RMSE: 0.15 ±0.00 -> 0.14 ±0.00 | Helped |
| use meteorological weather conditioning (ablation) | ConvLSTM, PredRNN, SimVP, and Contextformer · GreenEarthNet OOD-t | R2 | Helped |
| replace the PVT vision encoder with an MLP vision encoder (ablation) | Contextformer · GreenEarthNet OOD-t | R2: 0.62 -> 0.58 | Hurt |
| freeze the pre-trained PVT encoder (ablation) | Contextformer · GreenEarthNet OOD-t | Outperform Climatology: 62.3% -> 46.1% | Hurt |
| add delta prediction from the last historical cloud-free NDVI pixel (ablation) | Contextformer · GreenEarthNet OOD-t | Outperform Climatology: 62.3% -> 65.1% | Helped |
| use a learned MLP estimate instead of the last cloud-free pixel for initial vegetation state (ablation) | Contextformer · GreenEarthNet OOD-t | Outperform Climatology: 65.1% -> 60.6% | Hurt |
| add the cloud mask token on top of the delta-prediction scheme (ablation) | Contextformer · GreenEarthNet OOD-t | Outperform Climatology: 65.1% -> 66.8% | Helped |
| scale the model from 6M to 16M parameters (ablation) | Contextformer · GreenEarthNet OOD-t | Outperform Climatology: 66.8% -> 67.3% | Mixed |
| shuffle spatial inputs across batch and space (ablation) | Contextformer · GreenEarthNet OOD-t | R2: 0.62 -> 0.55 | Hurt |
| shuffle spatial inputs across batch and space (ablation) | PredRNN · GreenEarthNet OOD-t | R2: 0.62 -> 0.45 | Hurt |
| shuffle spatial inputs across batch and space (ablation) | SimVP · GreenEarthNet OOD-t | R2: 0.60 -> 0.49 | Hurt |
| shuffle spatial inputs across batch and space (ablation) | 1x1 LSTM · GreenEarthNet OOD-t | R2: 0.53 -> 0.53 | No clear effect |
Surprising findings
- Local time series models like Prophet outperform most deep video prediction baselines in outperformance score (60.6%), suggesting temporal dynamics dominate vegetation forecasting.
- Scaling Contextformer to 16M parameters does not improve performance when trained on GreenEarthNet.
Limitations
- Weather inputs at inference are historical observations rather than uncertain forecasts, so practical deployment is untested.
- Predictions oversmooth as lead time increases.
- Performance margin over climatology shrinks on spatial and spatio-temporal extrapolation tests.
Experimental setup
Datasets: GreenEarthNet EarthNet2021 CloudSEN12 ESA WorldCover E-OBS Geomorpho90m
Benchmarks: GreenEarthNet OOD-t GreenEarthNet OOD-s GreenEarthNet OOD-st
Metrics: R2 RMSE NSE |bias| Outperformance vs climatology RMSE over first 25 days
Baselines: Persistence Previous year Climatology Kalman filter LightGBM Prophet ConvLSTM SGED-ConvLSTM Earthformer PredRNN SimVP Next-frame UNet
Builds on: Pyramid Vision Transformer v2 EarthNet2021 Presto PredRNN SimVP
Closest papers by meaning
- EcoMapper: Generative Modeling for Climate-Aware Satellite Imagery (ICML 2025 · similarity 0.87)
- DiffusionSat: A Generative Foundation Model for Satellite Imagery (ICLR 2024 · similarity 0.85)
- SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (CVPR 2024 · similarity 0.85)
- GreenHyperSpectra: A multi-source hyperspectral dataset for global vegetation trait prediction (NeurIPS 2025 · similarity 0.85)
- Efficient Representation Learning of Satellite Image Time Series and Their Fusion for Spatiotemporal Applications (AAAI 2024 · similarity 0.85)
- AnySat: One Earth Observation Model for Many Resolutions, Scales, and Modalities (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.