Research map / Time series / Foundation models for time series
TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TEMPO, a GPT-2-based time series forecaster that decomposes input into trend, seasonal and residual components and adds semi-soft prompts, evaluated in a many-to-one zero-shot setting.
Problem
Time series forecasting with general-purpose pre-trained models that generalize zero-shot to unseen datasets, while capturing trend and seasonality patterns and leveraging contextual text.
Approach
Combine STL-style trend/seasonal/residual decomposition (each mapped to GPT tokens with learnable decomposition) with a semi-soft prompt (hard text template projected into learnable prompt vectors) on a GPT-2 backbone, adapting via LoRA and training on pooled source datasets; forecasts are the sum of component predictions.
Key results
- Zero-shot many-to-one long-term forecasting: TEMPO average MSE 0.216 on Electricity, 0.503 on Traffic, 0.282 on Weather, 0.501 on ETTm1 — best average vs. GPT2, T5, PatchTST, TimesNet among listed baselines.
- TEMPO outperforms baselines on TETS EBITDA and GDELT SMAPE/MSE in zero-shot cross-sector settings (e.g., GDELT event 11 SMAPE 38.77 vs 38.89 for BERT).
- Ablation: removing prompt or decomposition loss generally degrades results; prompt alone without decomposition can hurt performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TEMPO's decomposition and prompt-based pretrained GPT-2 instead of the compared forecasting baselines | Zero-shot long-term forecasting · ECL, Traffic, Weather, ETTm1, ETTm2, ETTh1, and ETTh2 | MSE: 0.216 vs 0.223; 0.503 vs 0.554; 0.282 vs 0.338; 0.501 vs 0.555; 0.280 vs 0.289; 0.428 vs 0.428; 0.361 vs 0.382 | Helped |
| Use TEMPO with contextual text embeddings and time-series inputs | TEMPO · EBITDA sectors CC, CD, Ind, RE and GDELT events 11, 17, 19 | SMAPE: 32.27/33.48 vs 32.65/33.83; 25.9/26.25 vs 26.34/26.62; 26.7/27.42 vs 27.17/27.98; 29.46/30.11 vs 29.63/30.48; 38.77 vs 3 | Helped |
| Use TEMPO for self-supervised representation learning, then fine-tune with 5% in-domain data | TEMPO · Weather (unseen-domain forecasting, Avg.) | MSE: 0.304 vs 0.328 | Helped |
| Remove prompt design (ablation) | TEMPO · Zero-shot forecasting across reported datasets | MSE and MAE | Hurt |
| Remove decomposition and prompt design (ablation) | TEMPO · Zero-shot forecasting across reported datasets | MSE and MAE | Hurt |
| Remove decomposition-loss alignment (ablation) | TEMPO · Zero-shot forecasting across reported datasets | MSE and MAE | Hurt |
| Use prompt design without decomposition (ablation) | TEMPO · Zero-shot forecasting across reported datasets | MSE and MAE | Hurt |
Surprising findings
- Prompt design without decomposition can detrimentally affect the backbone in most cases.
- Zero-shot TEMPO outperforms full-shot state-of-the-art models on ETTh1/ETTh2 in the appendix comparison, though these baseline numbers are taken from other papers.
Limitations
- Dependence on LLMs with better numerical reasoning may improve results (noted by authors).
- Pre-training on highly diverse domains may hinder pattern discernment; hyperparameter exploration limited to 3 or 6 GPT layers.
- TETS dataset and GDELT setup rely on ChatGPT-generated contextual summaries and SMAPE outlier filtering.
Experimental setup
Datasets: ETTh1 ETTh2 ETTm1 ETTm2 Weather Electricity Traffic GDELT TETS (S&P 500 EBITDA with ChatGPT-generated summaries)
Benchmarks: ETTh1 ETTh2 ETTm1 ETTm2 Weather Electricity Traffic TETS GDELT
Metrics: MSE MAE SMAPE
Baselines: GPT2 T5 BERT LLaMA PatchTST TimesNet FEDformer ETSformer Informer DLinear Reformer TimeGPT
Builds on: GPT-2 PatchTST N-BEATS L2P DualPrompt
Closest papers by meaning
- Timer: Generative Pre-trained Transformers Are Large Time Series Models (ICML 2024 · similarity 0.93)
- Time Series Generation Under Data Scarcity: A Unified Generative Modeling Approach (NeurIPS 2025 · similarity 0.91)
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.89)
- Large Pre-trained time series models for cross-domain Time series analysis tasks (NeurIPS 2024 · similarity 0.89)
- TimePerceiver: An Encoder-Decoder Framework for Generalized Time-Series Forecasting (NeurIPS 2025 · similarity 0.89)
- TimeCMA: Towards LLM-Empowered Multivariate Time Series Forecasting via Cross-Modality Alignment (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.