Research map / Time series / Long-term forecasting
Are Self-Attentions Effective for Time Series Forecasting?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CATS, a cross-attention-only time series transformer where horizon-dependent learnable queries attend to past patches, with heavy parameter sharing and query-adaptive masking, achieving competitive long-term forecasting with fewer parameters and less memory.
Problem
Whether self-attention is effective for time series forecasting, given that linear models have been reported to outperform Transformer forecasters and self-attention is permutation-invariant, losing temporal order information.
Approach
Remove all self-attention and use only cross-attention in which future-horizon-dependent learnable queries attend to embedded past patches as keys/values; share embedding, attention, and projection parameters across horizons, and apply query-adaptive masking (stochastic-depth-like) on the attention-to-LayerNorm connection.
Key results
- On long-term forecasting with input length 96, CATS achieves the lowest average MSE on Traffic across all four horizons (e.g., 0.421 at horizon 96) and best results on most Electricity and ETT horizons.
- On M4 short-term forecasting, CATS reports average SMAPE 11.701, MASE 1.557, OWA 0.838, slightly better than TimeMixer (11.723, 1.559, 0.840) and reduces MASE by 19.94% versus PatchTST.
- On ETTm1 with input length 2880, CATS keeps about 0.4M parameters while PatchTST grows to 33.6M and TimeMixer to 52.9M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace self-attention with a linear embedding layer (ablation) | PatchTST · forecasting horizons across datasets (all reported cases) | forecasting performance | Helped |
| Replace cross-attention layers with self-attention (one or two layers) (ablation) | CATS · Electricity and ETTm1, horizons 96–720 | MSE: Electricity: 0.126 -> 0.128/0.133; ETTm1 (720): 0.400 -> 0.429/0.442 | Hurt |
| Use CATS instead of TimeMixer | long-term forecasting models · Traffic, horizons 96–720 | MSE: 0.462 -> 0.421 (96); 0.506 -> 0.484 (720) | Helped |
| Use CATS instead of TimeMixer | long-term forecasting models · Weather, horizon 720 | MSE: 0.339 -> 0.342 | Hurt |
| Use CATS instead of TimeMixer | long-term forecasting models · Electricity, horizons 96–720 | MSE: 0.153 -> 0.149 (96); 0.225 -> 0.219 (720) | Helped |
| Use CATS instead of TimeMixer | short-term forecasting models · M4 Average | MASE: 1.559 -> 1.557 | Helped |
| Increase input sequence length from 336 to 2880 | CATS · ETTm1 | MSE: 0.407 -> 0.395 | Helped |
Surprising findings
- Removing self-attention from PatchTST improved or maintained performance across horizons, with the original self-attention version worse on longer horizons.
- Replacing cross-attention layers with self-attention in CATS mostly degraded performance.
Limitations
- Assumes channel independence, so cross-variate dependencies are not modeled.
- Reported gains on several benchmarks are small or mixed; CATS is not best on all horizons (e.g., Weather and ETTm2 at some horizons).
Experimental setup
Datasets: Electricity ETTh1 ETTh2 ETTm1 ETTm2 Weather Traffic M4
Benchmarks: Electricity ETTh1 ETTh2 ETTm1 ETTm2 Weather Traffic M4
Metrics: MSE MAE SMAPE MASE OWA parameter count GPU memory running time
Baselines: TimeMixer PatchTST TimesNet Crossformer MICN FiLM DLinear Autoformer Informer
Builds on: PatchTST DLinear Transformer stochastic depth
Closest papers by meaning
- A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle (ICML 2025 · similarity 0.89)
- Sequence Complementor: Complementing Transformers for Time Series Forecasting with Learnable Sequences (AAAI 2025 · similarity 0.89)
- DeformableTST: Transformer for Time Series Forecasting without Over-reliance on Patching (NeurIPS 2024 · similarity 0.87)
- SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention (ICML 2024 · similarity 0.87)
- TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables (NeurIPS 2024 · similarity 0.87)
- iTransformer: Inverted Transformers Are Effective for Time Series Forecasting (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.