Research map / Time series / Long-term forecasting
Timer-XL: Long-Context Transformers for Unified Time Series Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Timer-XL is a causal decoder-only Transformer for unified time series forecasting that generalizes next-token prediction to multivariate patch tokens via a Kronecker-structured TimeAttention mask and enlarged context.
Problem
Existing time series Transformers have short effective contexts and struggle to unify univariate, multivariate, and covariate-informed forecasting, especially for long-context inputs where encoder-only models degrade.
Approach
Multivariate next token prediction over flattened patch tokens of all variables, with TimeAttention masks built as a Kronecker product of a variable-dependency matrix and a causal temporal mask, plus RoPE for time and learnable scalars for permutation-equivalent variable handling.
Key results
- Timer-XL achieves best 96-pred-96 multivariate results on ECL (0.138 MSE), Traffic (0.387 MSE), and Weather (0.165 MSE), among compared models.
- On ERA5-S univariate forecasting with 3072-token input, Timer-XL averages 0.168 MSE versus 0.175 for PatchTST.
- In zero-shot forecasting, Timer-XL-Base wins 15 first-place counts overall, outperforming Time-MoE-Base and Moirai baselines with a comparable model size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Timer-XL instead of PatchTST | univariate forecasting · ERA5-S average, input-3072-pred-96 | MSE: 0.175 -> 0.168 | Helped |
| add instance normalization (ablation) | Timer-XL · ERA5-S average, input-3072-pred-96 | MSE: 0.168 -> 0.172 | Hurt |
| add inter-series modeling with TimeAttention | Timer · ECL, 96-pred-96 | MSE: 0.159 -> 0.138 | Helped |
| add inter-series modeling with TimeAttention | Timer · Traffic, 96-pred-96 | MSE: 0.413 -> 0.387 | Helped |
| use a longer lookback context | univariate forecasting models · univariate benchmarks | forecast accuracy | Mixed |
| replace the temporal causal mask with an all-one mask (ablation) | Timer-XL · EPF average, 168-pred-24 | MSE: 0.302 -> 0.316 | Hurt |
| use decoder-only rather than encoder-only architecture (ablation) | patched multivariate Transformer · ERA5-MS average, input-3072-pred-96 | MSE: 0.170 -> 0.166 | Helped |
| enable channel-dependent rather than channel-independent modeling (ablation) | patched encoder-only Transformer · ERA5-MS average, input-3072-pred-96 | MSE: 0.176 -> 0.170 | Helped |
| pre-train with Timer-XL instead of Chronos Large | zero-shot forecasting · ETTm2 | MSE: 0.295 -> 0.273 | Helped |
Surprising findings
- Encoder-only Transformers can degrade in long-context forecasting, while decoder-only models mitigate this degradation.
- Normalization-based gains on univariate benchmarks may stem from insufficient time span and training samples rather than being essential.
Limitations
- Performance on monthly and yearly contexts improves slowly or deteriorates, attributed to noise and training difficulty.
- Channel-dependent attention scales as O(N^2 T^2) without FlashAttention, limiting high-dimensional series.
- Multivariate forecasting with rolling evaluation and sufficient training samples is the regime where it helps most.
Experimental setup
Datasets: ETT Weather ECL Traffic Solar-Energy PEMS03 EPF GTWSF UTSD LOTSA ERA5-S ERA5-MS
Benchmarks: ECL ETTh1 Traffic Weather Solar-Energy ERA5-S ERA5-MS EPF GTWSF ETTm1 ETTm2 ETTh2
Metrics: MSE MAE
Baselines: PatchTST iTransformer UniTST TimeXer DLinear TimesNet Timer Moirai Time-MoE Chronos TimesFM MOMENT
Builds on: Timer PatchTST RoPE FlashAttention iTransformer
Closest papers by meaning
- A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle (ICML 2025 · similarity 0.89)
- TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables (NeurIPS 2024 · similarity 0.89)
- iTransformer: Inverted Transformers Are Effective for Time Series Forecasting (ICLR 2024 · similarity 0.89)
- Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization (ICML 2025 · similarity 0.88)
- Timer: Generative Pre-trained Transformers Are Large Time Series Models (ICML 2024 · similarity 0.88)
- Sequence Complementor: Complementing Transformers for Time Series Forecasting with Learnable Sequences (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.