Research map / Time series / Foundation models for time series
Unified Training of Universal Time Series Forecasting Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Moirai, a masked-encoder universal time series forecasting Transformer with multi-patch-size projections, any-variate attention, and a mixture-distribution head, trained on the 27B-observation LOTSA archive for zero-shot forecasting.
Problem
Building a single pre-trained (universal) forecasting model that handles cross-frequency data, arbitrary numbers of variates, and heterogeneous distributions, which the one-model-per-dataset paradigm cannot address.
Approach
A masked encoder Transformer with frequency-specific multi-patch-size input/output projections, Any-variate Attention (RoPE over time plus learned binary attention biases over variate identity on a flattened multivariate sequence), and a mixture of parametric distributions trained by NLL on a large open archive with random context/horizon sampling.
Key results
- On the Monash in-distribution benchmark, Moirai outperforms all Monash baselines at every size; the ablation shows the full model reaches normalized MAE 0.655 vs 0.904 without Any-variate Attention and 1.156 without multi patch size.
- As a zero-shot forecaster on probabilistic benchmarks, Moirai-Base and the large model obtain best or second-best CRPS on most datasets (e.g., Weather CRPS 0.041 for Moirai-Base vs 0.043 for TFT full-shot).
- Packing improves performance by 16% at constant compute, reducing padding from 61.08% to 0.38% of input tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| allow any frequency to use any patch size during training (ablation) | Moirai · Monash benchmark | normalized MAE: 0.655 -> 0.720 | Hurt |
| use a single patch size of 32 instead of multi patch sizes (ablation) | Moirai · Monash benchmark | normalized MAE: 0.655 -> 1.156 | Hurt |
| replace Any-variate Attention with additive learned embeddings (ablation) | Moirai · Monash benchmark | normalized MAE: 0.655 -> 0.904 | Hurt |
| replace the mixture distribution with a Student's t-distribution (ablation) | Moirai · Monash benchmark | normalized MAE: 0.655 -> 0.740 | Hurt |
| train on LOTSA rather than only GluonTS and Monash datasets (ablation) | Moirai · Monash benchmark | normalized MAE: 0.809 -> 0.655 | Helped |
| use sequence packing during training (ablation) | Moirai · Monash benchmark | normalized MAE: 0.785 -> 0.655 | Helped |
| use packed training instead of un-packed training at constant compute (ablation) | Moirai · training performance | performance improvement: 16% | Helped |
Surprising findings
- Padding accounted for 61.08% of input tokens without packing, a large inefficiency in flattened multivariate time series training.
- The relationship between model size and zero-shot performance on long sequence forecasting is tenuous; the large model is less consistent than the base model.
Limitations
- Little to no hyperparameter tuning was performed due to resource constraints.
- Multi patch size mapping for cross-frequency learning is heuristic.
- Limited support for high-dimensional time series.
Experimental setup
Datasets: LOTSA Monash GluonTS datasets BuildingsBench ClimateLearn (ERA5, CMIP6) CloudOps TSF LargeST LibCity ProEnFo SubseasonalClimateUSA Electricity Solar
Benchmarks: Monash Time Series Forecasting Benchmark Electricity Solar Walmart Weather Istanbul Traffic Turkey Power ETTh1 ETTh2 ETTm1 ETTm2 Electricity (LSF)
Metrics: normalized MAE CRPS MSIS MSE MAE
Baselines: DeepAR PatchTST TiDE TFT AutoARIMA Seasonal Naive iTransformer TimesNet Crossformer DLinear SCINet FEDformer
Builds on: Transformer RoPE SwiGLU RMSNorm PatchTST
Closest papers by meaning
- Towards a General Time Series Forecasting Model with Unified Representation and Adaptive Transfer (ICML 2025 · similarity 0.89)
- Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (ICLR 2025 · similarity 0.88)
- TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting (ICLR 2024 · similarity 0.88)
- FilterNet: Harnessing Frequency Filters for Time Series Forecasting (NeurIPS 2024 · similarity 0.88)
- SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention (ICML 2024 · similarity 0.88)
- Large Pre-trained time series models for cross-domain Time series analysis tasks (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.