Research map / Time series / Long-term forecasting

TSLANet: Rethinking Transformers for Time Series Representation Learning

ICML 2024 · Time series › Long-term forecasting · approach: Transformers for long-term forecasting

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TSLANet, a lightweight convolutional, Fourier-based universal architecture (Adaptive Spectral Block plus Interactive Convolution Block with masked-patch self-supervised pretraining) that competes with or beats Transformers on classification, forecasting, and anomaly detection.

Problem

Transformers for time series suffer from noise sensitivity, high computational cost, and overfitting on small datasets, while CNNs struggle with long-range dependencies in forecasting and anomaly detection.

Approach

Replace self-attention with an Adaptive Spectral Block that applies FFT-domain global and local learnable filters with a learnable power threshold to suppress high-frequency noise, followed by an Interactive Convolution Block where two kernel-size convolutions gate each other multiplicatively, plus masked-patch self-supervised pretraining.

Key results

• Averaged over classification subsets, TSLANet reaches 85.90% accuracy versus 83.55% for TS-TCC and 83.46% for ROCKET, with 83.18% on UCR and 72.73% on UEA.
• On anomaly detection, TSLANet averages 87.54% F1 versus 86.72% for GPT4TS, the second-best method.
• In forecasting, TSLANet is second-best to Time-LLM on most datasets; on ETTh1 it gets MSE 0.413 versus Time-LLM's 0.408, at 6.9e+10 vs 7.3e+12 FLOPS.

Measured effects reported in the paper

Surprising findings

• A simple single-layer MLP is reported as competitive on some UCR subsets, and TimesNet is noted to struggle on datasets with simpler structure.
• On forecasting, TSLANet does not win outright; Time-LLM, built on Llama-7B, is slightly better on most datasets.

Limitations

• Forecasting results trail Time-LLM, which uses a 7B LLM backbone, so TSLANet is not best on forecasting overall.
• Pretraining is per dataset rather than a single foundation model, which limits the foundation-model claim.
• Several ablation and efficiency numbers are reported on single datasets only.

Experimental setup

Datasets: UCR archive (85 datasets) UEA archive (26 datasets) Sleep-EDF MIT-BIH Arrhythmia UCIHAR WISDM HHAR ECL ETT Exchange Traffic Weather

Benchmarks: UCR UEA Sleep-EDF MIT-BIH UCIHAR WISDM HHAR ECL ETT Exchange Traffic Weather

Metrics: accuracy MSE MAE F1-score FLOPs

Baselines: GPT4TS TimesNet ROCKET TS-TCC TS2Vec Crossformer PatchTST iTransformer Time-LLM DLinear

Builds on: FNet / Fourier-domain learnable filters (GFNet) PatchTST masked autoencoder for time series

Closest papers by meaning

• T-Rep: Representation Learning for Time Series using Time-Embeddings (ICLR 2024 · similarity 0.87)
• TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysis (ICLR 2025 · similarity 0.87)
• TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation (ICML 2025 · similarity 0.87)
• SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention (ICML 2024 · similarity 0.86)
• Multi-Patch Prediction: Adapting Language Models for Time Series Representation Learning (ICML 2024 · similarity 0.86)
• TimeCHEAT: A Channel Harmony Strategy for Irregularly Sampled Multivariate Time Series Analysis (AAAI 2025 · similarity 0.86)