Research map / Time series / Foundation models for time series
Timer: Generative Pre-trained Transformers Are Large Time Series Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Timer pre-trains a decoder-only GPT-style Transformer by next-token prediction on a curated 1B-point time series corpus (UTSD) using a single-series sequence format, then adapts it to forecasting, imputation, and anomaly detection as a large time series model.
Problem
Small deep time series models degrade in data-scarce settings, and prior time series pre-training is small-scale, encoder-based, and task-specific; the paper asks how to build a scalable, task-general large time series model.
Approach
Unify heterogeneous time series into single-series sequences (S3) of fixed context length, tokenize them into segments, and pre-train a decoder-only Transformer autoregressively (GPT-style next-segment prediction), casting forecasting, imputation, and anomaly detection as generative tasks.
Key results
- Timer fine-tuned with about 1% of ETTh1 samples and 5% of Traffic outperforms advanced small forecasters trained on full data (per the paper's claim).
- Scaling Timer reduces few-shot PEMS forecasting MSE from 0.231 to 0.138 (-40.3%) and 0.194 to 0.123 (-36.6%), surpassing a multivariate forecaster trained on full PEMS (0.139).
- In zero-shot forecasting, Timer is ranked among the top LTSMs with Moirai and TimesFM on seven unseen datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train Timer on UTSD-12G | Timer decoder-only Transformer · PEMS (Avg), 1% Target | MSE: 0.328 -> 0.180 | Helped |
| pre-train Timer on UTSD-12G | Timer decoder-only Transformer · PEMS (Avg), 5% Target | MSE: 0.215 -> 0.138 | Helped |
| pre-train Timer on UTSD-12G | Timer decoder-only Transformer · PEMS (Avg), 20% Target | MSE: 0.153 -> 0.126 | Helped |
| fine-tune from the pre-trained Timer instead of training from scratch | Timer · Weather, full samples | MSE: 0.165→0.154 | Helped |
| fine-tune from the pre-trained Timer instead of training from scratch | Timer · PEMS03, full samples | MSE: 0.126→0.118 | Helped |
| fine-tune from the pre-trained Timer instead of training from scratch | Timer · PEMS04, full samples | MSE: 0.125→0.107 | Helped |
| increase model size and pre-training data scale | Timer · PEMS few-shot scenarios | MSE: 0.231→0.138 (-40.3%) and 0.194→0.123 (-36.6%) | Helped |
| increase the number of layers from 1M to 4M parameters (ablation) | Timer · PEMS forecasting, few-shot scenarios | forecasting error: 14.7% and 20.6% | Helped |
| increase model dimension from 3M to 50M parameters (ablation) | Timer · PEMS forecasting, few-shot scenarios | forecasting error: 25.1% and 18.2% | Helped |
| increase the pre-training dataset scale (ablation) | Timer · downstream forecasting | forecasting error | Helped |
Surprising findings
- Autoregressive decoder-only models pre-trained on large data can perform as competitively as direct multi-step predictors.
- Zero-shot performance shows only a weak positive correlation with pre-training scale among existing LTSMs.
Limitations
- Scaling evidence is limited to moderate model sizes (up to 50M parameters in the controlled study).
- Zero-shot correlation between scale and performance is weak, suggesting data quality matters more than scale.
Experimental setup
Datasets: UTSD-1G UTSD-2G UTSD-4G UTSD-12G LOTSA ETT ECL Traffic Weather PEMS UCR Anomaly Archive
Benchmarks: ETT ECL Traffic Weather PEMS UCR Anomaly Archive TimesNet imputation benchmark
Metrics: MSE number of detected anomalies under quantiles average rank
Baselines: PatchTST iTransformer TimesNet Anomaly Transformer TiDE TCN LSTM Moirai TimesFM LLMTime Lag-Llama
Builds on: GPT PatchTST TimesNet channel independence
Closest papers by meaning
- TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting (ICLR 2024 · similarity 0.93)
- Time Series Generation Under Data Scarcity: A Unified Generative Modeling Approach (NeurIPS 2025 · similarity 0.91)
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.90)
- Large Pre-trained time series models for cross-domain Time series analysis tasks (NeurIPS 2024 · similarity 0.89)
- Timer-XL: Long-Context Transformers for Unified Time Series Forecasting (ICLR 2025 · similarity 0.88)
- SynTSBench: Rethinking Temporal Pattern Learning in Deep Learning Models for Time Series (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.