Research map / Time series / Foundation models for time series
FSTLLM: Spatio-Temporal LLM for Few Shot Time Series Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FSTLLM, which uses LLM-derived node embeddings to build a graph for an STGNN and then QLoRA-fine-tunes LLaMA-2-7B on prompts containing STGNN predictions to improve few-shot spatio-temporal forecasting.
Problem
Spatio-temporal GNNs and time series foundation models need large training data and struggle in few-shot forecasting settings where only about a week of training data is available.
Approach
Use LLM embeddings of node-specific textual context to learn an adjacency matrix for an STGNN, then fine-tune an LLM with domain-rich prompts that include the STGNN's numerical predictions so the LLM refines them.
Key results
- On Nottingham, FSTLLM achieves best results in 22 of 24 MAE/RMSE/MAPE evaluations, with MAPE roughly 30% lower than baselines.
- On ECL, FSTLLM is best in 25 of 36 evaluations and in the top two in 32 of 36, with over 50% relative MAPE reduction versus GPT4TS and iTransformer.
- Plugging FSTLLM onto GPT4TS and iTransformer lowers Nottingham MAE from 27.6 to 21.8 and from 28.4 to 22.3 respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Domain Knowledge Injection to FSTLLM (ablation) | FSTLLM · Nottingham | MAE: 25.1 -> 21.0 | Helped |
| Add Domain Knowledge Injection to FSTLLM (ablation) | FSTLLM · Nottingham | MAPE: 23.9% -> 20.7% | Helped |
| Add Domain Knowledge Injection to FSTLLM (ablation) | FSTLLM · Nottingham | RMSE: 94.5 -> 82.3 | Helped |
| Use LLM-Enhanced Graph Construction and Domain Knowledge Injection instead of cosine similarity graph learning without injection (ablation) | FSTLLM · Nottingham | MAE: 27.1 -> 21.0 | Helped |
| Use LLM-Enhanced Graph Construction and Domain Knowledge Injection instead of cosine similarity graph learning without injection (ablation) | FSTLLM · Nottingham | MAPE: 25.1% -> 20.7% | Helped |
| Use LLM-Enhanced Graph Construction and Domain Knowledge Injection instead of cosine similarity graph learning without injection (ablation) | FSTLLM · Nottingham | RMSE: 98.0 -> 82.3 | Helped |
| Integrate FSTLLM with GPT4TS predictions | GPT4TS · Nottingham | MAE: 27.6 -> 21.8 | Helped |
| Integrate FSTLLM with iTransformer predictions | iTransformer · Nottingham | MAE: 28.4 -> 22.3 | Helped |
| Use FSTLLM instead of GTS | GTS · Nottingham, 15 mins | MAE: 7.89 -> 7.83 | Helped |
| Use FSTLLM instead of GTS | GTS · Nottingham, 15 mins | RMSE: 42.79 -> 47.93 | Hurt |
| Use FSTLLM instead of GraphWaveNet | GraphWaveNet · ECL, 1 hour | MAE: 1497 -> 1663 | Hurt |
| Use FSTLLM instead of iTransformer | iTransformer · ECL, 1 hour | RMSE: 2947 -> 3503 | Hurt |
Surprising findings
- Replacing the LLM component of several LLM-based forecasters with a simple sample layer does not degrade performance, per cited prior work.
- Time-MoE is reported to be less effective in few-shot settings due to its expert routing mechanism.
Limitations
- Evaluated on only two real-world datasets (Nottingham and ECL).
- Few-shot setting is defined as only the most recent week of training data, so generality to other data regimes is untested.
- Dependence on node-specific textual documents and user reviews that must be collected for each node.
Experimental setup
Datasets: Nottingham ECL
Benchmarks: Nottingham ECL
Metrics: MAE RMSE MAPE
Baselines: ARIMA VAR DCRNN GraphWaveNet STSGCN GMAN GTS PatchTST iTransformer DLinear GPT4TS TimeLLM
Builds on: GTS Time-LLM GPT4TS QLoRA alpha-Entmax
Closest papers by meaning
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.91)
- AutoTimes: Autoregressive Time Series Forecasters via Large Language Models (NeurIPS 2024 · similarity 0.88)
- ST-FiT: Inductive Spatial-Temporal Forecasting with Limited Training Data (AAAI 2025 · similarity 0.88)
- STEM-LTS: Integrating Semantic-Temporal Dynamics in LLM-driven Time Series Analysis (AAAI 2025 · similarity 0.87)
- TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning (NeurIPS 2025 · similarity 0.87)
- STD-PLM: Understanding Both Spatial and Temporal Properties of Spatial-Temporal Data with PLM (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.