Research map / Time series / Foundation models for time series
Efficient Representation Learning of Satellite Image Time Series and Their Fusion for Spatiotemporal Applications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PatchNet, a beam-search/attention-based partial spatial traversal encoder for satellite image time series, and FuSITSNet, a contrastive feature-level fusion of Landsat-8 and MODIS series for crop yield, snow cover, and solar energy prediction.
Problem
Processing high spatial resolution satellite image time series is computationally prohibitive, and single satellites trade off spatial against temporal resolution, so high-spatial and high-temporal data cannot be obtained together.
Approach
Learn a patch-selection mechanism that processes only about one-quarter of Landsat-8 patches (guided by attention and neighbor selection), then fuse the patch embeddings with MODIS embeddings at the feature level via patch alignment and cross-modality attention, avoiding generated intermediate images.
Key results
- FuSITSNet reduces corn yield RMSE from 21.47 (PatchNet on Landsat-8) to 16.19 bu/ac, and soybean from 7.29 to 5.04 bu/ac.
- PatchNet with about one-quarter (p=4) of the Landsat-8 patches outperforms the histogram-based CYN model on corn yield (21.47 vs 21.82 RMSE) without using soil or meteorological data.
- FuSITSNet cuts solar energy prediction RMSE from 12.81 (PatchNet Landsat-8) to 9.23, and beats GAN-enhanced PatchNet (11.95).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add patch alignment module (ablation) | FuSITSNet · CYP, corn | RMSE: 17.198 -> 16.192 | Helped |
| add patch alignment module (ablation) | FuSITSNet · CYP, soybean | RMSE: 5.286 -> 5.038 | Helped |
| add patch alignment module (ablation) | FuSITSNet · SCP | RMSE: 12.448 -> 9.230 | Helped |
| add patch alignment module (ablation) | FuSITSNet · SEP | RMSE: 2.501 -> 2.044 | Helped |
| add cross-modality attention (ablation) | FuSITSNet · CYP, corn | RMSE: 17.242 -> 16.192 | Helped |
| add cross-modality attention (ablation) | FuSITSNet · CYP, soybean | RMSE: 5.847 -> 5.038 | Helped |
| add cross-modality attention (ablation) | FuSITSNet · SCP | RMSE: 11.523 -> 9.230 | Helped |
| add cross-modality attention (ablation) | FuSITSNet · SEP | RMSE: 2.749 -> 2.044 | Helped |
| replace patch selection and neighbor selection with random patch selection (ablation) | PatchNet · CYP, corn | RMSE: 24.29 -> 21.47 | Helped |
| replace patch selection and neighbor selection with random patch selection (ablation) | PatchNet · CYP, soybean | RMSE: 9.98 -> 7.29 | Helped |
| add meteorological data | PatchNet · CYP, corn | RMSE: 21.469 -> 20.631 | Helped |
| add meteorological data | PatchNet · CYP, soybean | RMSE: 7.290 -> 6.963 | Helped |
Surprising findings
- Random patch selection in PatchNet yields much worse RMSE (24.29 vs 21.47 for corn), showing the learned selection matters.
Limitations
- Applicable only to permutation-invariant applications for now.
- Experiments are limited to Landsat-8 and MODIS over US counties.
Experimental setup
Datasets: Landsat-8 image time series MODIS image time series USDA Quick Stats crop yield labels MOD10A1 snow cover solar energy production data for 5 states
Benchmarks: crop yield prediction (corn, soybean) on US counties snow cover prediction solar energy prediction
Metrics: RMSE
Baselines: CNN (You et al. 2017) CNN+GP CNN+LSTM CYN STARFM RSFN GAN (Bouabid et al. 2020) TSE (MODIS)
Builds on: beam search STARFM margin contrastive loss
Closest papers by meaning
- Multi-modal Learning for Geospatial Vegetation Forecasting (CVPR 2024 · similarity 0.85)
- SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (CVPR 2024 · similarity 0.85)
- DiffusionSat: A Generative Foundation Model for Satellite Imagery (ICLR 2024 · similarity 0.85)
- Exact: Exploring Space-Time Perceptive Clues for Weakly Supervised Satellite Image Time Series Semantic Segmentation (CVPR 2025 · similarity 0.83)
- Galileo: Learning Global & Local Features of Many Remote Sensing Modalities (ICML 2025 · similarity 0.83)
- AnySat: One Earth Observation Model for Many Resolutions, Scales, and Modalities (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.