Research map / Time series / Long-term forecasting
Linear Transformers as VAR Models: Aligning Autoregressive Attention Mechanisms with Autoregressive Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that linear attention layers can be read as dynamic VAR models, rearranges MLP/attention/residual flow so stacked linear attention stays VAR-aligned, and proposes SAMoVAR, a linear Transformer for multivariate time series forecasting.
Problem
Deep autoregressive Transformers for time series forecasting structurally mismatch the autoregressive/VAR generative process, which obscures the VAR structure in linear attention and hurts interpretability and generalization.
Approach
A single linear attention layer is a dynamic VAR with keys as observations and outer products of value and query as weights; by reordering MLP, attention, and residual/input-output flow, stacked linear attention layers yield a mixture of VAR weights over temporal influence paths (SAMoVAR), with ARX patch tokenization and RMSNorm on queries and values.
Key results
- SAMoVAR achieves the best average rank (1.41) and 29 top-1 results out of 33 settings in the summarized multivariate table, versus CATS at 2.86 average rank.
- On PEMS04 (96 horizon) SAMoVAR reports MSE 0.092 versus 0.116 for the next best listed baseline, with over 30% gains on Solar and PEMS datasets as claimed.
- Ablations show that reintroducing W_k or removing RMSNorm on queries/values degrades ETTh1 MSE (0.401 to 0.413 and 0.421), and l=3 layers gives the best ETTh1 result (0.401).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SAMoVAR instead of the strongest reported baseline | Multivariate TSF models · Weather, Solar, ETTh1, ETTm1, ETTm2, PEMS03, PEMS04, PEMS08 | MSE: 0.217 -> 0.214 (Weather); 0.189 -> 0.184 (Solar); 0.419 -> 0.401 (ETTh1); 0.346 -> 0.339 (ETTm1); 0.243 -> 0.240 (ETTm2) | Helped |
| Use SAMoVAR instead of CATS | Multivariate TSF models · ETTh2; Traffic | MSE: 0.320 -> 0.324 (ETTh2); 0.385 -> 0.391 (Traffic) | Hurt |
| Add key projection weights W_k (ablation) | SAMoVAR · ETTh1 and ETTm1 | MSE: 0.401 -> 0.413 (ETTh1); 0.339 -> 0.346 (ETTm1) | Hurt |
| Remove inverse output matrix D^-1 (ablation) | SAMoVAR · ETTh1 and ETTm1 | MSE: 0.401 -> 0.409 (ETTh1); 0.339 -> 0.344 (ETTm1) | Hurt |
| Remove RMSNorm from queries and values (ablation) | SAMoVAR · ETTh1 and ETTm1 | MSE: 0.401 -> 0.421 (ETTh1); 0.339 -> 0.350 (ETTm1) | Hurt |
| Increase attention heads while reducing head dimension (ablation) | SAMoVAR · ETTh1 and ETTm1 | MSE: ETTh1: 0.401 -> 0.406 -> 0.412 -> 0.413; ETTm1: 0.339 -> 0.342 -> 0.343 -> 0.344 | Hurt |
| Use one attention layer instead of three (ablation) | SAMoVAR · ETTh1 and ETTm1 | MSE: 0.401 -> 0.420 (ETTh1); 0.339 -> 0.346 (ETTm1) | Hurt |
| Increase attention depth from three to four layers (ablation) | SAMoVAR · ETTh1 and ETTm1 | MSE: 0.401 -> 0.404 (ETTh1); 0.339 -> 0.345 (ETTm1) | Mixed |
Surprising findings
- Adding the key projection W_k, which seems natural, hurts performance and causes training instability.
- A single-layer model (l=1) performs worst because it lacks intermediate influence-path points.
Limitations
- Larger SAMoVAR models not tested on large-scale general TSF or as foundation models.
- Not evaluated on general sequence modeling tasks beyond TSF.
- Some baselines failed on PEMS07 under batch size 1 with L_I=4096, so that dataset is excluded from the main text.
Experimental setup
Datasets: Weather Solar Electricity (ECL) ETTh1 ETTh2 ETTm1 ETTm2 Traffic PEMS03 PEMS04 PEMS08 synthetic random VAR(p)
Benchmarks: Weather Solar ECL ETTh1 ETTh2 ETTm1 ETTm2 Traffic PEMS03 PEMS04 PEMS08
Metrics: MSE MAE average rank
Baselines: LinTrans FixedVAR CATS iTransformer FITS PatchTST DLinear Encformer
Builds on: linear attention PatchTST RevIN VARX
Closest papers by meaning
- WAVE: Weighted Autoregressive Varying Gate for Time Series Forecasting (ICML 2025 · similarity 0.90)
- iTransformer: Inverted Transformers Are Effective for Time Series Forecasting (ICLR 2024 · similarity 0.89)
- A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle (ICML 2025 · similarity 0.89)
- Revisiting Attention for Multivariate Time Series Forecasting (AAAI 2025 · similarity 0.89)
- Sequence Complementor: Complementing Transformers for Time Series Forecasting with Learnable Sequences (AAAI 2025 · similarity 0.88)
- HDT: Hierarchical Discrete Transformer for Multivariate Time Series Forecasting (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.