Research map / Time series / Long-term forecasting
DeformableTST: Transformer for Time Series Forecasting without Over-reliance on Patching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeformableTST, a hierarchical Transformer with a data-driven deformable sparse attention that reduces reliance on patching for time series forecasting.
Problem
Patch-based Transformer forecasters need very long inputs and large patches to perform well, which makes them unsuitable for tasks with short input lengths; full attention spreads focus over all time points without patching.
Approach
Replaces full attention with a learnable deformable sparse attention that samples important time points via input-dependent offsets, combined with a hierarchical backbone and optional small patching, so the model focuses on key time points without relying on patch partitions.
Key results
- Long-term forecasting averaged over input lengths 96/384/768: ETTh1 MSE 0.413 vs PatchTST-style baselines around 0.43-0.44 (Pathformer 0.439, CARD 0.430).
- Univariate short-term forecasting achieves averaged 14.1% SMAPE improvement over the second-best model.
- Multivariate short-term forecasting achieves averaged 25.6% MSE improvement over the second-best model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DeformableTST instead of the second-best model on univariate short-term forecasting | Univariate short-term forecasting tasks · M1, M3, M4, Tourism, NN5, Hospital, KDD Cup | SMAPE: averaged 14.1% SMAPE promotion | Helped |
| Use DeformableTST instead of the second-best model on multivariate short-term forecasting | Multivariate short-term forecasting tasks · Exchange, ILI, ETTh1, ETTh2, PEMS | MSE: averaged 25.6% MSE promotion | Helped |
| Add hierarchical structure after removing patching (ablation) | PatchTST adapted toward DeformableTST · Long-term forecasting | memory usage | Helped |
| Add deformable attention after removing patching (ablation) | PatchTST adapted toward DeformableTST · Long-term forecasting | MSE | Helped |
| Add local enhancement modules (LPU and ConvFFN) (ablation) | DeformableTST · Long-term forecasting | MSE | Helped |
| Replace deformable attention with ProbSparse attention (ablation) | DeformableTST · Long-term forecasting, input length 96; eight datasets | MSE: ETTh1: 0.425 -> 0.431; ETTh2: 0.346 -> 0.379; ETTm1: 0.373 -> 0.390; ETTm2: 0.283 -> 0.295; Weather: 0.251 -> 0.267; Sol | Hurt |
| Replace deformable attention with AutoCorrelation (ablation) | DeformableTST · Long-term forecasting, input length 96; eight datasets | MSE: ETTh1: 0.425 -> 0.449; ETTh2: 0.346 -> 0.375; ETTm1: 0.373 -> 0.393; ETTm2: 0.283 -> 0.292; Weather: 0.251 -> 0.263; Sol | Hurt |
| Replace deformable attention with FourierAttention (ablation) | DeformableTST · Long-term forecasting, input length 96; eight datasets | MSE: ETTh1: 0.425 -> 0.453; ETTh2: 0.346 -> 0.374; ETTm1: 0.373 -> 0.384; ETTm2: 0.283 -> 0.290; Weather: 0.251 -> 0.266; Sol | Hurt |
| Replace deformable attention with full attention (ablation) | DeformableTST · Long-term forecasting, input length 96; eight datasets | MSE: ETTh1: 0.425 -> 0.447; ETTh2: 0.346 -> 0.373; ETTm1: 0.373 -> 0.385; ETTm2: 0.283 -> 0.293; Weather: 0.251 -> 0.260; Sol | Hurt |
| Replace deformable attention with window attention (ablation) | DeformableTST · Long-term forecasting, input length 96; eight datasets | MSE: ETTh1: 0.425 -> 0.441; ETTh2: 0.346 -> 0.370; ETTm1: 0.373 -> 0.380; ETTm2: 0.283 -> 0.291; Weather: 0.251 -> 0.263; Sol | Hurt |
Surprising findings
- Full attention without patching spreads focus almost evenly over all input time points, and PatchTST's performance degrades without patching, suggesting patching is a crutch for attention.
Limitations
- Does not model multivariate correlations (variate-independent embedding), which the authors note hurts short-term multivariate performance on PEMS versus cross-variate methods like CARD and iTransformer.
- Patching is still used for very long inputs for efficiency, so reliance is reduced rather than eliminated.
Experimental setup
Datasets: Weather Traffic ECL Solar-Energy ETTh1 ETTh2 ETTm1 ETTm2 PEMS03 PEMS04 PEMS07 PEMS08
Benchmarks: Weather Traffic ECL Solar-Energy ETTh1 ETTh2 ETTm1 ETTm2 PEMS Exchange ILI M1
Metrics: MSE MAE SMAPE MASE OWA
Baselines: Pathformer CARD GPT4TS PatchTST iTransformer FEDformer Autoformer RLinear TiDE TimesNet DLinear SCINet
Builds on: PatchTST Deformable DETR RevIN Swin Transformer
Closest papers by meaning
- A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle (ICML 2025 · similarity 0.89)
- HDT: Hierarchical Discrete Transformer for Multivariate Time Series Forecasting (AAAI 2025 · similarity 0.89)
- Unlocking the Power of Patch: Patch-Based MLP for Long-Term Time Series Forecasting (AAAI 2025 · similarity 0.89)
- Revisiting the Generic Transformer: Deconstructing a Strong Baseline for Time Series Foundation Models (NeurIPS 2026 · similarity 0.88)
- iTransformer: Inverted Transformers Are Effective for Time Series Forecasting (ICLR 2024 · similarity 0.88)
- Sequence Complementor: Complementing Transformers for Time Series Forecasting with Learnable Sequences (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.