Research map / Time series / Foundation models for time series
Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Moirai-MoE, a time series foundation model that replaces Moirai's frequency-specific projections with a single projection and token-level sparse mixture-of-experts Transformer, using a clustering-based gating function and decoder-only training on LOTSA.
Problem
Unified pretraining on heterogeneous time series is hard; existing foundation models impose frequency-level specialization (Moirai, TimesFM), which is too coarse and misaligned with actual time series patterns.
Approach
Use a single input/output projection and let a sparse mixture of experts in each Transformer block learn token-level specialization automatically, with a new gating function that assigns tokens by Euclidean distance to k-means centroids of a pretrained Moirai model's representations.
Key results
- Moirai-MoE-S beats Moirai-S by 17% on in-distribution Monash aggregate MAE and outperforms Moirai-B and Moirai-L by 8% and 7%.
- Moirai-MoE-B achieves the best zero-shot average CRPS (0.478) and MASE (0.651) across 10 datasets, ahead of TimesFM and Chronos.
- Moirai-MoE uses up to 65x fewer activated parameters than competing foundation models while matching or exceeding their accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace multi-frequency projection layers with single projection + sparse MoE (token-level specialization) (ablation) | Moirai-MoE-S vs Moirai-S · Monash in-distribution (29 datasets) | Aggregated MAE: 0.78 -> 0.65 with decoder-only and MoE combined | Helped |
| switch objective from masked encoder to decoder-only without MoE (ablation) | Moirai-S multi-projection · Monash in-distribution (29 datasets) | Aggregated MAE: 0.78 -> 0.75 | Helped |
| add MoE with single projection (on top of decoder-only) (ablation) | Moirai-MoE-S · Monash in-distribution (29 datasets) | Aggregated MAE: 0.75 -> 0.65 | Helped |
| token-cluster gating (k-means centroids from pretrained Moirai) instead of linear projection gating (ablation) | MoE gating · Monash in-distribution | Aggregated MAE / performance ranking: consistently superior across expert counts | Helped |
| decoder-only training objective instead of masked encoder (ablation) | Moirai-MoE training · Monash training steps | Aggregated MAE: decoder-only 50k steps matches masked encoder 100k steps | Helped |
| Moirai-MoE-S vs Moirai-S (single projection + MoE) | zero-shot forecasting · Zero-shot (10 datasets) | Average CRPS: 0.578 -> 0.497 | Helped |
| Moirai-MoE-B vs TimesFM and Chronos in zero-shot | zero-shot forecasting · Zero-shot (10 datasets) | Average CRPS and MASE: CRPS 0.478 vs TimesFM 0.488; MASE 0.651 vs Chronos-B 0.656 | Helped |
| increase patch size from 16 to 4 or other values (ablation) | Moirai-MoE · Monash in-distribution | Performance and inference speed: patch size 16 best; patch size 4 takes over a day to evaluate | Hurt |
| masking ratio 0.3 vs 0.1 or 0.5 (ablation) | Moirai-MoE decoder-only training · Monash in-distribution | Performance: 0.3 best; 0.1 and 0.5 worse | No clear effect |
Surprising findings
- Expert selection converges in deeper layers so that by the final layer only 3 of 32 experts are chosen, and allocation becomes nearly identical across frequencies, contrary to typical LLM MoE layer patterns.
- Changing the training objective alone from masked encoder to decoder-only gives only modest gains; most improvement comes from MoE token-level specialization.
Limitations
- Some experts are rarely selected; pruning left for future work.
- Largest Moirai-MoE-L was not trained due to compute requirements.
- Zero-shot evaluation covers only 10 datasets and Monash in-distribution uses datasets seen in pretraining corpus LOTSA (train splits).
Experimental setup
Datasets: LOTSA Monash benchmark (29 datasets for in-distribution evaluation)
Benchmarks: Monash benchmark (29 datasets) zero-shot forecasting suite (10 datasets)
Metrics: aggregated MAE normalized by seasonal naive (geometric mean) CRPS MASE
Baselines: Moirai TimesFM Chronos TiDE PatchTST iTransformer LLMTime Seasonal Naive
Builds on: Moirai PatchTST Switch Transformer Chronos
Closest papers by meaning
- Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (ICLR 2025 · similarity 0.89)
- MOMENT: A Family of Open Time-series Foundation Models (ICML 2024 · similarity 0.87)
- Unified Training of Universal Time Series Forecasting Transformers (ICML 2024 · similarity 0.86)
- Large Pre-trained time series models for cross-domain Time series analysis tasks (NeurIPS 2024 · similarity 0.86)
- Empowering Time Series Analysis with Large-Scale Multimodal Pretraining (NeurIPS 2026 · similarity 0.86)
- Federated Foundation Models on Heterogeneous Time Series (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.