Research map / Time series / Long-term forecasting
WAVE: Weighted Autoregressive Varying Gate for Time Series Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WAVE attention, which adds an implicit moving-average (MA) term to autoregressive attention variants via indirect MA weight generation, and shows that a decoder-only AR Transformer with ARMA structure is competitive with or beats SOTA TSF baselines.
Problem
Autoregressive decoder-only Transformers have been underexplored for end-to-end time series forecasting, and existing AR attention (exponential decay / gated linear attention) struggles to decouple short-term local effects from long-term seasonal patterns.
Approach
Introduce an ARMA structure into existing AR attention mechanisms by generating MA weights indirectly from linear-attention-style products over residuals (B matrix mapped to Theta = B(I-B)^{-1}), keeping O(N) complexity and parameter count via weight sharing.
Key results
- WAVE linear attention with ARMA reaches an average test MSE ranking of 2.333 (AvgRank) and 25 top-1 finishes across 25 settings in the short-term table, versus 12.813 for the worst baseline.
- Adding the MA term consistently lowers MSE for every attention variant; e.g., on ETTm1 (L_P short-term) standard attention drops from 0.300 to 0.239-type range improvements, and gated linear attention improves from 0.407 to 0.260.
- With L_I scaled to 4096, pure AR/WAVE Transformers keep improving while baselines degrade, and WAVE gains persist across layer counts m=1 to 8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the ARMA moving-average term to Linear Attention (ablation) | Linear Attention Transformer · 12 TSF datasets, LP 12/24/48/96, LI=512 | AvgRank: 4.271 -> 2.333 | Helped |
| Add the ARMA moving-average term to Standard Softmax Attention (ablation) | Standard Softmax Attention Transformer · 12 TSF datasets, LP 12/24/48/96, LI=512 | AvgRank: 7.958 -> 4.292 | Helped |
| Add the ARMA moving-average term to Gated Linear Attention (ablation) | Gated Linear Attention Transformer · 12 TSF datasets, LP 12/24/48/96, LI=512 | AvgRank: 12.375 -> 6.229 | Helped |
| Add the ARMA moving-average term to Element-wise Linear Attention (ablation) | Element-wise Linear Attention Transformer · 12 TSF datasets, LP 12/24/48/96, LI=512 | AvgRank: 8.938 -> 7.125 | Helped |
| Add the ARMA moving-average term to Fixed Attention (ablation) | Fixed Attention Transformer · 12 TSF datasets, LP 12/24/48/96, LI=512 | AvgRank: 11.146 -> 8.688 | Helped |
| Use WAVE instead of pure AR attention (ablation) | Autoregressive Transformer · 12 TSF datasets, LP 12/24/48/96 | MSE | Helped |
| Use Linear Attention instead of Standard Softmax Attention | Pure AR Transformer · Time series forecasting | MSE | Helped |
| Use RMSNorm instead of LayerNorm (ablation) | WAVE Transformer · Time series forecasting | Performance | No clear effect |
| Apply extra loss weighting to the last token (ablation) | Autoregressive Transformer · ETT datasets | Performance | Mixed |
Surprising findings
- Plain linear attention without denominator outperforms standard softmax attention for TSF.
- Gated linear attention alone underperforms linear attention on TSF, since the forget gate removes stable seasonal information.
Limitations
- Channel-independent design does not model inter-series relationships.
- Only evaluated on time series forecasting; generality to NLP not tested.
- Baselines are rerun with a modified AdamW setup, so their numbers may differ from originals.
Experimental setup
Datasets: Weather Solar Electricity (ECL) ETTh1 ETTh2 ETTm1 ETTm2 Traffic PEMS03 PEMS04 PEMS07 PEMS08
Benchmarks: Weather Solar ECL ETTh1 ETTh2 ETTm1 ETTm2 Traffic PEMS03 PEMS04 PEMS07 PEMS08
Metrics: MSE MAE average rank top-1 count
Baselines: FITS iTransformer CATS PatchTST DLinear MEGA Encoder-only Transformer
Builds on: linear attention gated linear attention Attention Free Transformer PatchTST RevIN MEGA
Closest papers by meaning
- Linear Transformers as VAR Models: Aligning Autoregressive Attention Mechanisms with Autoregressive Forecasting (ICML 2025 · similarity 0.90)
- Revisiting Attention for Multivariate Time Series Forecasting (AAAI 2025 · similarity 0.88)
- Sequence Complementor: Complementing Transformers for Time Series Forecasting with Learnable Sequences (AAAI 2025 · similarity 0.87)
- Are Self-Attentions Effective for Time Series Forecasting? (NeurIPS 2024 · similarity 0.87)
- Auto-Regressive Moving Diffusion Models for Time Series Forecasting (AAAI 2025 · similarity 0.86)
- AutoTimes: Autoregressive Time Series Forecasters via Large Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.