Research map / Label-efficient and robust learning / Self-supervised and contrastive
ST-ReP: Learning Predictive Representations Efficiently for Spatial-Temporal Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ST-ReP, a lightweight reconstruction-plus-prediction self-supervised pre-training model with a linear compression-extraction-decompression spatial-temporal encoder and multi-scale loss, evaluated by ridge regression on frozen representations across six spatial-temporal datasets.
Problem
Self-supervised representation learning for spatial-temporal series forecasting struggles with false negatives in contrastive learning due to variable homogeneity, ignores inter-variable spatial correlations, and has poor efficiency and scalability on large numbers of variables.
Approach
Non-contrastive reconstruction-based pre-training that jointly reconstructs current values and predicts unseen future values, with a Compression-Extraction-Decompression encoder that compresses time to a few latent steps, applies proxy-tensor linear attention across variables, and decompresses back, plus a multi-scale average-pooling loss applied only in the loss.
Key results
- On PEMS04 and PEMS08 at horizon 12, ST-ReP improves average MSE/MAE by 19.97%/11.25% over the second-best self-supervised baseline.
- ST-ReP achieves the highest average rank (1.4 at horizon 12) among nine self-supervised methods, but ranks worse than STEP at horizons 48 and 96 on several datasets.
- ST-ReP is the only self-supervised method that runs on the CA dataset (8600 nodes) without OOM, with 23.58 GB footprint and 458.5 s/epoch.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the ST-Encoder to the static context embedding model (ablation) | ST-ReP · spatial-temporal forecasting datasets | predictive performance | Helped |
| Add future-value prediction and its loss constraint (ablation) | ST-ReP · spatial-temporal forecasting datasets | predictive performance | Helped |
| Add current-value reconstruction and its loss constraint (ablation) | ST-ReP · spatial-temporal forecasting datasets | predictive performance | Helped |
| Add multi-scale temporal loss (ablation) | ST-ReP · spatial-temporal forecasting datasets | predictive performance | Helped |
| Increase temporal compression size p (ablation) | ST-ReP · spatial-temporal forecasting | prediction error | Mixed |
| Vary spatial proxy tensor size m (ablation) | ST-ReP · spatial-temporal forecasting | prediction error | No clear effect |
| Use ST-ReP rather than the strongest self-supervised baseline | PEMS04 and PEMS08 · PEMS04 and PEMS08, horizon 12 | MSE and MAE: +19.97%/11.25% MSE/MAE average improvement | Helped |
| Use ST-ReP rather than the strongest self-supervised baseline | Temperature and Humidity · climate datasets, horizon 12 | MSE and MAE | Mixed |
| Use ST-ReP rather than STEP | ST-ReP · PEMS04 and PEMS08, horizons 48 and 96 | forecasting performance | Hurt |
Surprising findings
- Without the ST-Encoder, the model stays competitive with the best baseline, suggesting the static embeddings and loss provide most of the supervision.
- On climate datasets ST-ReP is second to T-Rep, which the authors attribute to trend dominance over spatial dependencies.
Limitations
- Worse than STEP at long horizons (48 and 96), attributed to STEP's longer two-week input window during pre-training.
- Evaluation relies on linear ridge regression probing on frozen representations, so results may not transfer to stronger end-to-end downstream models.
- Performance gains on climate datasets are limited; the model is second to T-Rep there.
Experimental setup
Datasets: PEMS04 PEMS08 CA SDWPF Temperature Humidity
Benchmarks: PEMS04 PEMS08 CA SDWPF Temperature Humidity
Metrics: MSE MAE
Baselines: History Last Ridge Regression DLinear iTransformer TS2Vec CoST T-Rep TimesURL PatchTST ST-SSL STEP GPT-ST
Builds on: T-Rep ST-SSL STEP GPT-ST iTransformer
Closest papers by meaning
- How Different from the Past? Spatio-Temporal Time Series Forecasting with Self-Supervised Deviation Learning (NeurIPS 2025 · similarity 0.88)
- STD-PLM: Understanding Both Spatial and Temporal Properties of Spatial-Temporal Data with PLM (AAAI 2025 · similarity 0.87)
- ST-FiT: Inductive Spatial-Temporal Forecasting with Limited Training Data (AAAI 2025 · similarity 0.87)
- T-Rep: Representation Learning for Time Series using Time-Embeddings (ICLR 2024 · similarity 0.87)
- On the Integration of Spatial-Temporal Knowledge: A Lightweight Approach to Atmospheric Time Series Forecasting (NeurIPS 2025 · similarity 0.87)
- Causal Spatio-Temporal Prediction: An Effective and Efficient Multi-Modal Approach (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.