Research map / Time series / Long-term forecasting
iTransformer: Inverted Transformers Are Effective for Time Series Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Inverts the Transformer so each variate's whole lookback series becomes one token, applying self-attention across variates and the feed-forward network along time, achieving state-of-the-art long-term multivariate forecasting.
Problem
Transformer-based time series forecasters embed all variates at the same timestamp into one temporal token, which wipes out multivariate correlations, is too local, and degrades with longer lookback windows; linear models have recently outperformed them.
Approach
Keep the vanilla Transformer components unchanged but invert their roles: each variate's full time series is embedded as a variate token, self-attention models cross-variate correlations, and the shared feed-forward network learns series-level representations and decodes future values.
Key results
- iTransformer achieves best average results on ECL (MSE 0.178), Traffic (MSE 0.428), Weather (MSE 0.258), Solar-Energy (MSE 0.233) and PEMS (MSE 0.119) with lookback 96.
- Inverting the architecture improves vanilla Transformer by an average 38.9% MSE reduction, Reformer 36.1%, Informer 28.5%, Flowformer 16.8%, and Flashformer 32.2%.
- Ablation shows the vanilla Transformer arrangement (attention on temporal tokens, FFN on time) performs worst among designs, e.g. Traffic MSE 0.913 vs 0.428 for iTransformer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| invert the Transformer dimensions, using variate attention and temporal FFN | Transformer forecasters · ECL, Traffic, Weather | MSE: 0.277 -> 0.178; 0.665 -> 0.428; 0.657 -> 0.258 | Helped |
| replace temporal FFN with attention (ablation) | iTransformer · ECL, Traffic, Weather, Solar-Energy | MSE: 0.178 -> 0.193; 0.428 -> 0.913; 0.258 -> 0.255; 0.233 -> 0.261 | Hurt |
| swap variate attention and temporal FFN (ablation) | iTransformer · ECL, Traffic, Weather, Solar-Energy | MSE: 0.178 -> 0.202; 0.428 -> 0.863; 0.258 -> 0.258; 0.233 -> 0.285 | Hurt |
| replace variate attention with FFN (ablation) | iTransformer · ECL, Traffic, Weather, Solar-Energy | MSE: 0.178 -> 0.182; 0.428 -> 0.599; 0.258 -> 0.248; 0.233 -> 0.269 | Mixed |
| remove temporal FFN (ablation) | iTransformer · ECL, Traffic, Weather, Solar-Energy | MSE: 0.178 -> 0.189; 0.428 -> 0.456; 0.258 -> 0.261; 0.233 -> 0.258 | Hurt |
| remove variate attention (ablation) | iTransformer · ECL, Traffic, Weather, Solar-Energy | MSE: 0.178 -> 0.193; 0.428 -> 0.461; 0.258 -> 0.265; 0.233 -> 0.261 | Hurt |
| increase lookback length | iTransformer · time-series forecasting | MSE | Helped |
| train with randomly selected variates in each batch | iTransformer · high-dimensional multivariate forecasting | forecasting performance | No clear effect |
| use FFN on both dimensions instead of variate attention (ablation) | iTransformer · Weather | MSE: 0.258 -> 0.248 | Helped |
Surprising findings
- Applying attention on temporal tokens (the vanilla Transformer arrangement) performs worst among the component arrangements, while inverting gives consistent gains across Transformer variants.
- The inverted model improves with longer lookback windows, whereas vanilla Transformer-based forecasters do not necessarily benefit from them.
Limitations
- Attention cost grows quadratically with number of variates, making it slower than temporal-token Transformers on datasets with few variates and long lookbacks.
- Results are limited to forecasting tasks; large-scale pretraining is left for future work.
- Layer-wise improvements are dataset-dependent (e.g., Exchange where DLinear or RLinear are competitive).
Experimental setup
Datasets: ECL ETT Exchange Traffic Weather Solar-Energy PEMS03 PEMS04 PEMS07 PEMS08 Market
Benchmarks: ECL ETT Exchange Traffic Weather Solar-Energy PEMS
Metrics: MSE MAE
Baselines: PatchTST Crossformer TiDE TimesNet DLinear SCINet RLinear FEDformer Stationary Autoformer
Builds on: Transformer PatchTST DLinear Channel Independence Layer normalization
Closest papers by meaning
- A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle (ICML 2025 · similarity 0.90)
- SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention (ICML 2024 · similarity 0.90)
- OLinear: A Linear Model for Time Series Forecasting in Orthogonally Transformed Domain (NeurIPS 2025 · similarity 0.89)
- Linear Transformers as VAR Models: Aligning Autoregressive Attention Mechanisms with Autoregressive Forecasting (ICML 2025 · similarity 0.89)
- TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables (NeurIPS 2024 · similarity 0.89)
- Timer-XL: Long-Context Transformers for Unified Time Series Forecasting (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.