Research map / Video / Video segmentation and tracking
STD-PLM: Understanding Both Spatial and Temporal Properties of Spatial-Temporal Data with PLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STD-PLM, a GPT-2-based framework that uses spatial and temporal tokenizers, topology-aware node embeddings, and a sandglass attention module with a constrained loss to perform spatial-temporal forecasting and imputation with few-shot and zero-shot transfer.
Problem
Spatial-temporal forecasting and imputation are usually handled by separate task-specific models that need lots of training data and generalize poorly to zero-shot and few-shot settings; existing PLM-based spatial-temporal methods ignore temporal tokens, topology, and the cost of many spatial tokens
Approach
Tokenize spatial-temporal data into per-node spatial tokens plus overall state/trend temporal tokens, feed them through a partially fine-tuned GPT-2 (first three layers), and compress node tokens into region tokens with a sandglass attention (precoder/decoder) regularized by a graph-structure loss and a Dirichlet prior loss.
Key results
- Forecasting on PEMS08: STD-PLM MAE 13.31, RMSE 23.19, MAPE 8.84%, best among listed baselines including PDFormer (13.58 MAE).
- Imputation on PEMS08 RM 70%: MAE 14.36 vs. best baseline mTAN 21.23.
- Zero-shot PEMS04→PEMS08 gives MAE 29.52; with 5% training samples few-shot results are comparable to LSTM trained on full data (as claimed in text).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add temporal tokens (ablation) | STD-PLM · PEMS08 | MAE: 13.81 -> 13.31 | Helped |
| Add temporal tokens (ablation) | STD-PLM · PEMS08 CM 70% | MAE: 22.82 -> 22.69 | Helped |
| Add constrained loss (ablation) | STD-PLM · PEMS08 | MAE: 13.53 -> 13.31 | Helped |
| Add constrained loss (ablation) | STD-PLM · PEMS08 CM 70% | MAE: 23.62 -> 22.69 | Helped |
| Use a fine-tuned PLM instead of a same-scale transformer encoder (ablation) | STD-PLM · PEMS08 | MAE: 13.63 -> 13.31 | Helped |
| Use a fine-tuned PLM instead of a same-scale transformer encoder (ablation) | STD-PLM · PEMS08 CM 70% | MAE: 24.05 -> 22.69 | Helped |
| Use SGA rather than removing SGA (ablation) | STD-PLM · PEMS03 | inference time (s): 17.96 -> 7.40 | Helped |
| Use SGA rather than removing SGA (ablation) | STD-PLM · PEMS07 | inference time (s): 52.82 -> 9.15 | Helped |
Surprising findings
- Forecasting-only PLM baselines OFA, STGLLM and STLLM underperform conventional spatial-temporal models, which the authors take as evidence that naive PLM use is insufficient.
Limitations
- Only GPT-2 (first three layers) was tested; larger PLMs were not explored due to compute limits.
- No large-scale spatial-temporal pretraining was done, so emergent behavior is untested.
- Zero-shot and few-shot results are limited to four traffic datasets with sparse graphs.
Experimental setup
Datasets: PEMS03 PEMS04 PEMS07 PEMS08
Benchmarks: PEMS03 PEMS04 PEMS07 PEMS08
Metrics: MAE RMSE MAPE
Baselines: LSTM ASTGCN AGCRN SSTBAN PDFormer iTransformer OFA STGLLM STLLM BRITS E2GAN mTAN
Builds on: GPT-2 PatchTST LoRA CSDI SSTBAN
Closest papers by meaning
- ST-FiT: Inductive Spatial-Temporal Forecasting with Limited Training Data (AAAI 2025 · similarity 0.89)
- ST-ReP: Learning Predictive Representations Efficiently for Spatial-Temporal Forecasting (AAAI 2025 · similarity 0.87)
- Learning from Highly Sparse Spatio-temporal Data (NeurIPS 2024 · similarity 0.87)
- FSTLLM: Spatio-Temporal LLM for Few Shot Time Series Forecasting (ICML 2025 · similarity 0.87)
- Graph Structure Learning for Spatial-Temporal Imputation: Adapting to Node and Feature Scales (AAAI 2025 · similarity 0.86)
- How Different from the Past? Spatio-Temporal Time Series Forecasting with Self-Supervised Deviation Learning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.