Research map / Time series / Long-term forecasting
A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes why point-wise transformers underperform and why simple patch-wise/variate-wise transformers win on forecasting benchmarks, using a mutual-information-based metric and synthetic plus healthcare datasets.
Problem
Understanding why point-wise transformers are less competitive for time series forecasting, why intra- and inter-variate attention models perform similarly, and which architectural components drive the success of simple transformers on standard benchmarks.
Approach
Introduces model-agnostic mutual-information-style scores (Intra MI, Inter MI) estimated from input perturbations and output standard deviation, plus controlled synthetic datasets with tunable autocorrelation and inter-variate dependency, to show benchmarks are dominated by intra-variate, stationary self-dependence.
Key results
- Intra-variate MI correlates strongly negatively with MAE on benchmarks (e.g., -0.875 on ETTh1 column in Table 2), while inter-variate MI has little effect on performance.
- Removing skip connections from iTransformer raises Electricity MAE from 0.266 to 0.320 and Traffic MAE from 0.283 to 0.591.
- Z-score normalization substantially improves benchmark performance but degrades performance on non-stationary synthetic data with monotonic trends.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add encoder skip connections (ablation) | iTransformer · benchmark datasets, especially Electricity and Traffic | MAE: 0.320 -> 0.266 (Electricity); 0.591 -> 0.283 (Traffic) | Helped |
| add encoder skip connections (ablation) | iTransformer · Synthetic datasets with lower self-dependency or stronger inter-variate dependency | MAE: 0.764 vs 0.765 (γ = 0.5, α = 0.8) | No clear effect |
| use a variate-dependent decoder (ablation) | iTransformer · Synthetic (γ = 0.5), α = 0.8 | MAE: 0.765 -> 0.761 | Helped |
| use a variate-dependent decoder (ablation) | iTransformer · TIHM | MAE: 0.623 -> 0.610 | Helped |
| use a variate-dependent decoder (ablation) | iTransformer · MINDER | MAE: 0.419 -> 0.431 | Hurt |
| remove Z-score normalization and denormalization (ablation) | iTransformer · benchmark datasets such as Weather and ETTh2 | MAE: 0.299 -> 0.28 (Weather); 0.668 -> 0.407 (ETTh2) | Hurt |
| remove Z-score normalization and denormalization (ablation) | iTransformer · Synthetic (γ = 0.5), α = 0.8 | MAE: 0.765 -> 0.751 | Helped |
| remove Z-score normalization and denormalization (ablation) | iTransformer · TIHM | MAE: 0.634 -> 0.622 | Hurt |
| remove Z-score normalization and denormalization (ablation) | iTransformer · TIHM | MSE: 0.834 vs 0.835 | No clear effect |
Surprising findings
- Z-score normalization, not architecture, drives much of the success of PatchTST and iTransformer on benchmarks, and it hurts on non-stationary synthetic data.
- Models with inter-variate attention (iTransformer, TimeXer) show lower inter-variate MI than other models, indicating inter-variate patterns contribute little on benchmarks.
Limitations
- Synthetic datasets contain only two variates and are simple autoregressive processes.
- Real-world validation limited to two dementia-monitoring healthcare datasets.
- Mutual information is approximated via output standard deviation under Gaussian assumptions, a proxy rather than exact MI.
Experimental setup
Datasets: Weather Electricity Traffic ETTh1 ETTh2 ETTm1 ETTm2 MINDER TIHM synthetic datasets
Benchmarks: Weather Electricity Traffic ETTh1 ETTh2 ETTm1 ETTm2 MINDER TIHM synthetic datasets
Metrics: MAE MSE Intra MI Avg Inter MI Max Inter MI correlation between MAE and MI scores
Baselines: Transformer Autoformer FEDformer Crossformer PatchTST iTransformer TimeXer
Builds on: Transformer PatchTST iTransformer RevIN
Closest papers by meaning
- iTransformer: Inverted Transformers Are Effective for Time Series Forecasting (ICLR 2024 · similarity 0.90)
- SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention (ICML 2024 · similarity 0.90)
- TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables (NeurIPS 2024 · similarity 0.90)
- Sequence Complementor: Complementing Transformers for Time Series Forecasting with Learnable Sequences (AAAI 2025 · similarity 0.89)
- DeformableTST: Transformer for Time Series Forecasting without Over-reliance on Patching (NeurIPS 2024 · similarity 0.89)
- Timer-XL: Long-Context Transformers for Unified Time Series Forecasting (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.