Research map / Time series / Long-term forecasting
Sequence Complementor: Complementing Transformers for Time Series Forecasting with Learnable Sequences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sequence Complementors, learnable extra tokens concatenated to patch embeddings in transformers for time series forecasting, plus a volume-based diversification loss, improving PatchTST and iTransformer.
Problem
Transformers for time series forecasting learn low-diversity latent representations with limited training data, which correlates with higher forecasting error.
Approach
Concatenate K learnable sequences (complementors) to the patch embeddings so self-attention can draw on complementary information, and encourage their orthogonality with a differentiable log-volume (diversification) loss; the paper argues this raises representation entropy and lowers an MMSE bound.
Key results
- On long-term forecasting over 8 datasets, the method ranks first in 47 of the MSE/MAE settings versus 19 for iTransformer (the second-best).
- On M4 short-term forecasting, it achieves the best result in 14 of 15 settings (weighted OWA 0.836 vs 0.851 for the next best).
- Adding 3 complementors to iTransformer reduces average MSE by about 2.4% (e.g., ETTm1 0.407 to 0.393).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add complementary sequences (K ≥ 3) to PatchTST (ablation) | PatchTST · Exchange, Weather, ETTm1, and ETTh1 | forecasting error | Helped |
| use 3 complementary sequences instead of other tested counts (ablation) | Sequence Complementor · Exchange, Weather, ETTm1, and ETTh1 | forecasting error | Mixed |
| add diversification loss to complementary sequences (ablation) | Sequence Complementor · Exchange, Weather, ETTm1, and ETTh1 | forecasting error | Helped |
| add 3 complementary sequences to iTransformer (ablation) | iTransformer · ETTm1 | MSE: 0.407 -> 0.393 | Helped |
| add 3 complementary sequences to iTransformer (ablation) | iTransformer · ETTm2 | MSE: 0.288 -> 0.285 | Helped |
| add 3 complementary sequences to iTransformer (ablation) | iTransformer · ETTh1 | MSE: 0.454 -> 0.443 | Helped |
| add 3 complementary sequences to iTransformer (ablation) | iTransformer · ETTh2 | MSE: 0.383 -> 0.376 | Helped |
| add 3 complementary sequences to iTransformer (ablation) | iTransformer · Exchange | MSE: 0.36 -> 0.346 | Helped |
| add 3 complementary sequences to iTransformer (ablation) | iTransformer · Weather | MSE: 0.258 -> 0.254 | Helped |
Surprising findings
- Representation entropy and dominant-singular-value ratio show a strong negative correlation with MSE across transformer variants.
- On the ILI dataset, the method's average MSE (2.152) is worse than iTransformer's (2.039).
Limitations
- Channel-independent strategy only; cross-variate information not modeled.
- Diversity metrics are hard to compare across different feature dimensions and limited sample counts.
- Theoretical bounds rely on a Gaussian assumption.
Experimental setup
Datasets: ETTm1 ETTm2 ETTh1 ETTh2 Electricity Traffic Weather Exchange ILI M4
Benchmarks: ETTm1 ETTm2 ETTh1 ETTh2 Electricity Traffic Weather Exchange ILI M4
Metrics: MSE MAE SMAPE MASE OWA
Baselines: iTransformer RLinear PatchTST Crossformer TimesNet DLinear SCINet FEDformer Stationary (Non-stationary Transformer) Autoformer
Builds on: PatchTST RevIN Gaussian entropy estimation
Closest papers by meaning
- Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting (NeurIPS 2024 · similarity 0.90)
- A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle (ICML 2025 · similarity 0.89)
- Are Self-Attentions Effective for Time Series Forecasting? (NeurIPS 2024 · similarity 0.89)
- TimePerceiver: An Encoder-Decoder Framework for Generalized Time-Series Forecasting (NeurIPS 2025 · similarity 0.89)
- iTransformer: Inverted Transformers Are Effective for Time Series Forecasting (ICLR 2024 · similarity 0.88)
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.