Research map / Time series / Long-term forecasting
SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAMformer, a shallow one-layer channel-wise-attention transformer with RevIN trained under sharpness-aware minimization, which matches or beats MLP and transformer forecasters on 8 long-term forecasting benchmarks.
Problem
Transformers underperform simple linear and MLP baselines in multivariate long-term time series forecasting, and overfit or converge to sharp minima with poor generalization.
Approach
Use a shallow single-layer transformer with channel-wise (D x D) attention plus RevIN, and train it with sharpness-aware minimization to reach flatter minima; the paper argues attention entropy collapse is benign under SAM, and that sigma-reparameterization harms attention rank.
Key results
- SAMformer improves over TSMixer+SAM by 5.25% average MSE and over standalone TSMixer by 14.33%, with about 4x fewer parameters than TSMixer.
- SAMformer improves over the base Transformer by 16.96% average MSE and over PatchTST by 11.13%.
- SAMformer is on par with MOIRAI, improving overall MSE by 6.9% over MOIRAI-Small, 1.1% over MOIRAI-Base and 7.6% over MOIRAI-Large in the zero-shot comparison table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the transformer with SAM (ablation) | Transformer on the synthetic linear forecasting task · Synthetic validation data | validation loss | Helped |
| Fix self-attention weights at initialization (ablation) | Transformer on the synthetic linear forecasting task · Synthetic validation data | validation loss | Helped |
| Use sigmaReparam instead of the base Transformer (ablation) | Transformer on the synthetic linear forecasting task · Synthetic validation data | validation loss | Mixed |
| Use channel-wise rather than temporal-wise attention (ablation) | SAMformer · Multivariate forecasting benchmarks | test MSE | Helped |
| Replace learnable attention with identity attention (ablation) | SAMformer · Multivariate forecasting benchmarks | test MSE | Hurt |
| Add sigmaReparam to SAMformer (ablation) | SAMformer · ETTh1 and Exchange | test MSE | Hurt |
| Use sigmaReparam without SAM (ablation) | Transformer · ETTh1 and Exchange | test MSE | Hurt |
| Train TSMixer with SAM (ablation) | TSMixer · Multivariate forecasting benchmarks | test MSE: 9.58% | Helped |
| Use SGD instead of Adam (ablation) | SAMformer · Multivariate forecasting benchmarks | convergence | Hurt |
| Use AdamW instead of Adam (ablation) | SAMformer · Multivariate forecasting benchmarks | test performance | No clear effect |
| Increase SAM neighborhood size sufficiently, generally above 0.7 (ablation) | SAMformer · Multivariate forecasting benchmarks | test MSE: 0.7 | Helped |
| Combine SAM with sigmaReparam (ablation) | Transformer · Synthetic and real-world forecasting | test performance | No clear effect |
Surprising findings
- Sigma-reparameterization fails to close the gap to the oracle on the toy problem and on real data, even underperforming the base Transformer in some cases.
- Entropy collapse of attention appears benign under SAM, since SAM-trained attention keeps entropy close to the base Transformer while sharpness drops by orders of magnitude.
Limitations
- Exchange dataset at horizon 720 and some horizons show SAMformer not best (e.g., iTransformer and PatchTST win on Exchange)
- Sensitivity to the SAM neighborhood size rho requires tuning per dataset and horizon
- Baseline numbers for several models are taken from prior papers, some without RevIN
Experimental setup
Datasets: ETTh1 ETTh2 ETTm1 ETTm2 Electricity Exchange Traffic Weather
Benchmarks: ETTh1 ETTh2 ETTm1 ETTm2 Electricity Exchange Traffic Weather
Metrics: MSE MAE overall MSE improvement
Baselines: TSMixer TSMixer+SAM Transformer iTransformer PatchTST FEDformer Informer Autoformer MOIRAI sigma Reparam
Builds on: sharpness-aware minimization RevIN TSMixer sigma reparameterization entropy collapse analysis of transformers
Closest papers by meaning
- iTransformer: Inverted Transformers Are Effective for Time Series Forecasting (ICLR 2024 · similarity 0.90)
- A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle (ICML 2025 · similarity 0.90)
- HDT: Hierarchical Discrete Transformer for Multivariate Time Series Forecasting (AAAI 2025 · similarity 0.88)
- Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting (ICLR 2024 · similarity 0.88)
- DeformableTST: Transformer for Time Series Forecasting without Over-reliance on Patching (NeurIPS 2024 · similarity 0.88)
- Unified Training of Universal Time Series Forecasting Transformers (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.