Research map / Time series / Long-term forecasting
Revisiting Attention for Multivariate Time Series Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FSatten (frequency-spectrum attention with multi-head spectrum scaling) and SOatten (learnable orthogonal embedding with head-coupling convolution) as drop-in replacements for conventional attention in Variate and Temporal Transformers for multivariate time series forecasting.
Problem
Conventional attention in Transformers for multivariate time series forecasting computes dependencies in a learned linear latent space, and it is unclear whether this mapping space is optimal for capturing inter-sequence dependencies.
Approach
Replace the learned Q/K linear projection with a Fourier amplitude-spectrum representation scaled per head (MSS), and generalize to a learnable orthogonal embedding with a head-coupling 2D convolution over attention weights that exploits neighboring-variate similarity.
Key results
- SOatten(V) achieves average MSE 0.166 on ECL vs 0.178 for iTransformer, with FSatten reaching 0.162 on ECL.
- Across six benchmarks, FSatten and SOatten reportedly outperform the SOTA conventional-attention method by 8.1% and 21.8% on MSE overall.
- Replacing conventional attention with SOatten in PatchTST lowers average Traffic MSE from 0.555 to 0.492.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace conventional attention with SOatten in the Variate Transformer | iTransformer · ETTm1, ETTm2, ETTh1, ETTh2, ECL, Exchange, Weather, Solar-Energy | MSE: 0.407 -> 0.394; 0.288 -> 0.287; 0.454 -> 0.447; 0.383 -> 0.379; 0.178 -> 0.166; 0.360 -> 0.359; 0.258 -> 0.245; 0.233 -> | Helped |
| Replace conventional attention with SOatten in the Variate Transformer | iTransformer · Traffic | MSE: 0.428 -> 0.437 | Hurt |
| Replace conventional attention with FSatten in the Variate Transformer | iTransformer · ETTm1, ETTm2, ETTh1, ETTh2, ECL, Exchange, Weather, Solar-Energy | MSE: 0.407 -> 0.394; 0.288 -> 0.286; 0.454 -> 0.446; 0.383 -> 0.381; 0.178 -> 0.162; 0.360 -> 0.363; 0.258 -> 0.249; 0.233 -> | Helped |
| Replace conventional attention with FSatten in the Variate Transformer | iTransformer · Traffic | MSE: 0.428 -> 0.477 | Hurt |
| Replace PatchTST attention with SOatten | PatchTST · ETTm1, ETTm2, ETTh1, ETTh2, ECL, Exchange, Traffic, Weather, Solar-Energy | MSE: 0.387 -> 0.380; 0.281 -> 0.280; 0.469 -> 0.451; 0.387 -> 0.366; 0.216 -> 0.199; 0.367 -> 0.360; 0.555 -> 0.492; 0.259 -> | Helped |
| Replace MSS with a linear Query/Key projection (ablation) | FSatten and SOatten · Long-term MTSF benchmarks | MSE | Hurt |
| Add Head Coupling Convolution (HCC) (ablation) | SOatten · Long-term MTSF benchmarks, especially Temporal Transformer | MSE | Helped |
| Use a 3×3 HCC kernel rather than other tested kernel sizes (ablation) | SOatten · Temporal and Variate Transformers | MSE | Helped |
Surprising findings
- FSatten is weaker than the conventional attention baseline on Traffic (0.477 vs 0.428 MSE for iTransformer), and its performance varies widely across datasets.
Limitations
- FSatten is not well suited to Temporal Transformers because sub-series from one variate share periodic frequencies.
- SOatten's orthogonality is only enforced at initialization, not during backpropagation.
- Experiments are limited to a single input length of 96 and mainstream Temporal and Variate Transformers.
Experimental setup
Datasets: ECL ETTm1 ETTm2 ETTh1 ETTh2 Exchange Traffic Weather Solar-Energy
Benchmarks: ECL ETTm1 ETTm2 ETTh1 ETTh2 Exchange Traffic Weather Solar-Energy
Metrics: MSE MAE
Baselines: iTransformer PatchTST Crossformer TiDE TimesNet DLinear SCINet FEDformer Autoformer
Builds on: iTransformer PatchTST Transformer attention
Closest papers by meaning
- Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting (NeurIPS 2024 · similarity 0.91)
- Linear Transformers as VAR Models: Aligning Autoregressive Attention Mechanisms with Autoregressive Forecasting (ICML 2025 · similarity 0.89)
- WAVE: Weighted Autoregressive Varying Gate for Time Series Forecasting (ICML 2025 · similarity 0.88)
- CATS: Enhancing Multivariate Time Series Forecasting by Constructing Auxiliary Time Series as Exogenous Variables (ICML 2024 · similarity 0.88)
- Revitalizing Multivariate Time Series Forecasting: Learnable Decomposition with Inter-Series Dependencies and Intra-Series Variations Modeling (ICML 2024 · similarity 0.87)
- TACTiS-2: Better, Faster, Simpler Attentional Copulas for Multivariate Time Series (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.