Research map / Time series / Foundation models for time series
MOMENT: A Family of Open Time-series Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MOMENT, open encoder-only T5-style time series foundation models pre-trained with masked patch reconstruction on the compiled Time Series Pile, evaluated across five tasks under limited supervision.
Problem
Pre-training large time series models is hard due to lack of a large cohesive public time series corpus, heterogeneous series characteristics that complicate multi-dataset training, and a lack of benchmarks for evaluating foundation models under limited resources and supervision.
Approach
Compile a large public collection (Time Series Pile) and pre-train T5-sized transformer encoders from random initialization on fixed-length (512) univariate series split into patches, using masked patch reconstruction with reversible instance normalization and channel-independent handling; evaluate with zero-shot, linear-probing, and unsupervised-representation protocols across five tasks.
Key results
- Linear-probed MOMENT is second only to PatchTST on most long-horizon forecasting settings in MSE.
- Zero-shot MOMENT SVM on UCR representations outperforms all but 4 of 17 compared methods in mean accuracy-based comparisons (mean accuracy 0.794 vs TimesNet 0.572).
- MOMENT with linear probing achieves the highest mean adjusted best F1 (0.628) on UCR anomaly archive, outperforming TimesNet (0.537) and GPT4TS (0.424).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use linear probing rather than zero-shot reconstruction | MOMENT-Large · UCR Anomaly Archive | Mean adjusted best F1: 0.585 -> 0.628 | Helped |
| Use linear probing rather than zero-shot reconstruction | MOMENT-Large · ETTh1 imputation | MSE: 0.402 -> 0.139 | Helped |
| Use linear probing rather than zero-shot reconstruction | MOMENT-Large · ETTm2 imputation | MSE: 0.078 -> 0.031 | Helped |
| Use MOMENTLP instead of the strongest displayed baseline, TS2Vec | MOMENT-Large · UCR classification (91 datasets) | Mean accuracy: 0.851 vs 0.794 | Hurt |
| Use MOMENTLP instead of k-NN | MOMENT-Large · UCR Anomaly Archive | Mean VUS-ROC: 0.684 vs 0.706 | Hurt |
| Use MOMENTLP instead of PatchTST | MOMENT-Large · Weather, 96-step long-horizon forecasting | MSE: 0.154 vs 0.149 | Hurt |
| Initialize MOMENT-small randomly instead of with Flan-T5 weights (ablation) | MOMENT-small pre-training · Time Series Pile pre-training | Training loss | Helped |
| Increase model size | MOMENT pre-training · Time Series Pile pre-training | Training loss | Helped |
Surprising findings
- Statistical methods (Theta, ETS, ARIMA) and N-BEATS often outperform deeper transformer and LLM-based baselines on several tasks.
- Randomly initialized MOMENT reaches lower pre-training loss than initializing from language-model (Flan-T5) weights given enough data.
Limitations
- Zero-shot short-horizon forecasting lags statistical methods such as ARIMA and Theta.
- Cannot distinguish vertically shifted series because of per-series normalization.
- Fixed input length of 512 and channel-independent modeling limit multivariate and long-context use.
Experimental setup
Datasets: Time Series Pile Informer long-horizon forecasting datasets Monash forecasting archive UCR/UEA classification archive TSB-UAD anomaly benchmark UCR anomaly archive
Benchmarks: Informer long-horizon forecasting datasets M3 M4 UCR classification archive UCR anomaly archive ETT/Electricity/Weather imputation
Metrics: MSE MAE sMAPE accuracy adjusted best F1 VUS-ROC
Baselines: GPT4TS TimesNet PatchTST Time-LLM N-BEATS DLinear FEDFormer TS2Vec ARIMA Anomaly Transformer DGHL k-NN
Builds on: TimesNet benchmark T5 PatchTST Reversible instance normalization Time-LLM
Closest papers by meaning
- Empowering Time Series Analysis with Large-Scale Multimodal Pretraining (NeurIPS 2026 · similarity 0.89)
- Large Pre-trained time series models for cross-domain Time series analysis tasks (NeurIPS 2024 · similarity 0.88)
- In-Context Fine-Tuning for Time-Series Foundation Models (ICML 2025 · similarity 0.88)
- SEMPO: Lightweight Foundation Models for Time Series Forecasting (NeurIPS 2025 · similarity 0.88)
- Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysis (NeurIPS 2024 · similarity 0.87)
- OATS: Online Data Augmentation for Time Series Foundation Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.