Research map / Time series / Foundation models for time series
SEMPO: Lightweight Foundation Models for Time Series Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEMPO, a 6.5M-parameter time series foundation model pre-trained on ~83M time points, combining energy-aware spectral decomposition with a mixture-of-prompts Transformer for zero- and few-shot forecasting.
Problem
Existing time series foundation models are very large and need massive pre-training data, and they under-utilize low-energy frequency signals, which limits deployment in resource-constrained settings.
Approach
Split each series' spectrum into high- and low-energy branches using a learnable energy threshold, apply independently parameterized multi-band frequency masks per branch, and adapt a small Transformer via a token-routed mixture of dataset-specific prompts (prefix-style key/value prompts) instead of large MoE layers.
Key results
- Zero-shot on TSLib with 6.5M params and 83M pre-training points: best MSE on 12 of 14 dataset-metric cells (e.g., ETTh2 MSE 0.341 vs 0.355 for next best), reporting average MSE reductions of 23.1% over all methods.
- Few-shot (5% training data) on TSLib: SEMPO achieves best MSE on 12 cells, average MSE reduction of 32.3% versus compared models.
- Ablation: replacing dual-branch masking with multi-band masking raises average zero-shot MSE from 0.350 to 0.399.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SEMPO instead of the strongest compared models | Zero-shot forecasting · TSLib | MSE: 23.1% | Helped |
| Use SEMPO instead of the strongest compared models | Few-shot forecasting with 5% training data · TSLib | MSE: 32.3% | Helped |
| Replace dual-branch energy-aware spectral masking with multi-band masking (ablation) | SEMPO · TSLib, zero-shot average | MSE: 0.350 -> 0.399 | Hurt |
| Replace dual-branch spectral masking with random patch masking (ablation) | SEMPO · TSLib, zero-shot average | MSE: 0.350 -> 0.416 | Hurt |
| Replace MoP with sparse MoE (3 experts, 1 activated) (ablation) | SEMPO · TSLib, zero-shot | MSE | Hurt |
| Remove MoP's adaptive router and use prefix tuning (ablation) | SEMPO · TSLib, zero-shot | MSE | Hurt |
| Increase the number of masked series | SEMPO · ETTh1/h2, ETTm1/m2, Weather | forecasting performance | Mixed |
| Vary the prompt number | SEMPO · ETTh1/h2, ETTm1/m2, Weather | forecasting performance | No clear effect |
| Increase latent dimension beyond 256 | SEMPO · ETTh1/h2, ETTm1/m2, Weather | forecasting performance | Hurt |
Surprising findings
- Lightweight TTM reaches comparable MSE on ETTh1 with only 1M parameters and 3.2s inference, far faster than SEMPO (22s), which the authors acknowledge is less competitive on efficiency.
Limitations
- Forecasting is univariate via channel independence; multivariate interactions are not modeled.
- Less competitive than TTM on parameter count and inference time, per the authors' efficiency table.
- Performance declines at larger model sizes, likely from mismatch between capacity and the 83M-point pre-training data.
Experimental setup
Datasets: UTSD subset (~83M time points) ETTh1 ETTh2 ETTm1 ETTm2 Weather Electricity Traffic GIFT-Eval
Benchmarks: TSLib (ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity, Traffic) GIFT-Eval (9 datasets)
Metrics: MSE MAE SMAPE NRMSE
Baselines: TTM Time-MoE Timer Moirai Chronos TimesFM Moment Time-LLM GPT4TS S2IP-LLM iTransformer DLinear
Builds on: prefix tuning PatchTST Mixture of Experts RMSNorm SwiGLU
Closest papers by meaning
- Less is More: Unlocking Specialization of Time Series Foundation Models via Structured Pruning (NeurIPS 2025 · similarity 0.89)
- TimeBase: The Power of Minimalism in Efficient Long-term Time Series Forecasting (ICML 2025 · similarity 0.88)
- Towards a General Time Series Forecasting Model with Unified Representation and Adaptive Transfer (ICML 2025 · similarity 0.88)
- Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (ICLR 2025 · similarity 0.88)
- MOMENT: A Family of Open Time-series Foundation Models (ICML 2024 · similarity 0.88)
- TimePFN: Effective Multivariate Time Series Forecasting with Synthetic Data (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.