Research map / Time series / Long-term forecasting
Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Spectral Attention (SA) and Batched Spectral Attention (BSA), a plug-in module that keeps EMA low-pass features across sequentially ordered training samples and attends over frequency-filtered activations, letting fixed look-back forecasters exploit long-range trends.
Problem
Fixed-size look-back windows and shuffled mini-batch training prevent time series forecasters from using long-range temporal dependencies that span thousands of steps.
Approach
Store exponential moving averages of intermediate activations at multiple smoothing factors as low-pass filters across consecutive samples, then learn an attention matrix over low-pass and high-pass components; unfolding the EMA over a mini-batch lets gradients flow back across samples like BPTT. The module is identity-initialized and fine-tuned on top of a pretrained base model.
Key results
- BSA improves average MSE for all 7 base models, with gains ranging from about 0.96% to 7.2% across 11 datasets; e.g., DLinear average MSE 0.640 to 0.614.
- Statistically significant (paired t-test p<0.05) improvements in 82% of dataset-model cases per the authors' report; all MAE improvements in aggregate table are significant for most models.
- On synthetic ETTh1 with added 300-period sine wave, iTransformer improvement is nearly 30% across all prediction lengths.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add BSA to DLinear | DLinear · 11 datasets, Avg. | MSE: 0.640 -> 0.614 | Helped |
| Add BSA to RLinear | RLinear · 11 datasets, Avg. | MSE: 0.664 -> 0.619 | Helped |
| Add BSA to FreTS | FreTS · 11 datasets, Avg. | MSE: 0.807 -> 0.783 | Helped |
| Add BSA to TimesNet | TimesNet · 11 datasets, Avg. | MSE: 0.611 -> 0.605 | Helped |
| Add BSA to iTransformer | iTransformer · 11 datasets, Avg. | MSE: 0.551 -> 0.545 | Helped |
| Add BSA to Crossformer | Crossformer · 11 datasets, Avg. | MSE: 0.828 -> 0.814 | Helped |
| Add BSA to PatchTST | PatchTST · 11 datasets, Avg. | MSE: 0.524 -> 0.501 | Helped |
| Add BPTT across samples within the mini-batch (ablation) | BSA on iTransformer · Weather, 720-step prediction | MSE: 0.3480 -> 0.3401 | Helped |
| Add BPTT across samples within the mini-batch (ablation) | BSA on iTransformer · Weather, 720-step prediction | MAE: 0.3452 -> 0.3452 | No clear effect |
| Add multiple smoothing factors to BPTT-enabled BSA (ablation) | BSA on iTransformer · Weather, 720-step prediction | MSE: 0.3401 -> 0.3320 | Helped |
| Add multiple smoothing factors to BPTT-enabled BSA (ablation) | BSA on iTransformer · Weather, 720-step prediction | MAE: 0.3452 -> 0.3395 | Helped |
| Make smoothing factors learnable (ablation) | BSA on iTransformer with BPTT and multiple smoothing factors · Weather, 720-step prediction | MSE: 0.3320 -> 0.3263 | Helped |
Surprising findings
- Synthetic experiments show BSA improves beyond a 96-step look-back for 1000-period signals, and SA-matrix shifts toward low frequency only when long-range trends exist.
- Ablation shows BSA's effect comes from BPTT-style gradient flow; without it, performance is similar to non-batched SA.
Limitations
- Placement of BSA within the base model was not analyzed in detail; only a few insertion sites tested.
- Gains may be limited for datasets whose look-back window already contains only high-frequency information.
- Some large-model results (e.g., full Illness/ETTh2 runs) were incomplete at submission time.
Experimental setup
Datasets: Weather Traffic ECL ETTh1 ETTh2 ETTm1 ETTm2 Exchange PEMS03 EnergyData Illness
Benchmarks: Weather Traffic ECL ETTh1 ETTh2 ETTm1 ETTm2 Exchange PEMS03 EnergyData Illness
Metrics: MSE MAE percent improvement over base model paired t-test p-value
Baselines: DLinear RLinear FreTS TimesNet iTransformer Crossformer PatchTST
Builds on: DLinear iTransformer PatchTST RevIN
Closest papers by meaning
- Revisiting Attention for Multivariate Time Series Forecasting (AAAI 2025 · similarity 0.91)
- Sequence Complementor: Complementing Transformers for Time Series Forecasting with Learnable Sequences (AAAI 2025 · similarity 0.90)
- TimeBase: The Power of Minimalism in Efficient Long-term Time Series Forecasting (ICML 2025 · similarity 0.87)
- FilterNet: Harnessing Frequency Filters for Time Series Forecasting (NeurIPS 2024 · similarity 0.87)
- Are Self-Attentions Effective for Time Series Forecasting? (NeurIPS 2024 · similarity 0.87)
- Unlocking the Power of Patch: Patch-Based MLP for Long-Term Time Series Forecasting (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.