Research map / Time series / Foundation models for time series
Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Time-MoE, a sparse mixture-of-experts decoder-only time series foundation model scaled to 2.4B parameters and pre-trained on the 300B-point Time-300B corpus.
Problem
Pre-trained time series foundation models are small and expensive at inference; scaling them for universal forecasting while keeping compute and flexibility (arbitrary context and horizon) is unresolved.
Approach
A decoder-only transformer with point-wise tokenization and sparse MoE FFN layers (one shared expert plus top-k routed experts), trained with multi-resolution forecasting heads and a greedy scheduler for arbitrary horizons, so that activated parameters and inference cost stay small while total capacity grows.
Key results
- Zero-shot average MSE of 0.322 for Time-MoE-ultra versus 0.349 for Moirai-small and 0.359 for Moirai-large (about 20%+ MSE reduction across baselines).
- Fine-tuned Time-MoE reduces average MSE by about 24% relative to full-shot deep models, per the paper's claim.
- Sparse MoE reduces training cost by 78% and inference cost by 39% versus equivalent dense models, per the paper.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale the model from competitive time-series foundation models to Time-MoE ultra | Time-MoE · Six zero-shot benchmarks, average | MSE: 0.347 -> 0.322 | Helped |
| fine-tune Time-MoE on downstream training data for one epoch | Time-MoE · Six in-distribution benchmarks, average | MSE: achieving a MSE reduction of 24% in average | Helped |
| add Huber loss (ablation) | Time-MoE base · Six benchmarks, horizon 96 | MSE: 0.267 -> 0.262 | Helped |
| add multi-resolution forecasting layers and multi-task optimization (ablation) | Time-MoE base · Six benchmarks, horizon 96 | MSE: 0.269 -> 0.262 | Helped |
| use MoE layers instead of standard FFNs (ablation) | Time-MoE base · Six benchmarks, horizon 96 | MSE: 0.272 -> 0.262 | Helped |
| add auxiliary load-balancing loss (ablation) | Time-MoE base · Six benchmarks, horizon 96 | MSE: 0.275 -> 0.262 | Helped |
| use forecast horizons {1,8,32,64} rather than fewer output projections (ablation) | Time-MoE base · Six benchmarks, horizon 96 | MSE: 0.273; 0.320; 1.382 vs 0.262 | Helped |
| activate Top1 experts instead of Top2 (ablation) | Time-MoE base · Six benchmarks, horizon 96 | MSE: 0.262 -> 0.264 | Hurt |
| activate Top4 experts instead of Top2 (ablation) | Time-MoE base · Six benchmarks, horizon 96 | MSE: 0.262 -> 0.262 | No clear effect |
| activate Top6 experts instead of Top2 (ablation) | Time-MoE base · Six benchmarks, horizon 96 | MSE: 0.262 -> 0.265 | Hurt |
| activate Top8 experts instead of Top2 (ablation) | Time-MoE base · Six benchmarks, horizon 96 | MSE: 0.262 -> 0.269 | Hurt |
| use FP32 instead of BF16 precision (ablation) | Time-MoE base · Six benchmarks, horizon 96 | MSE: 0.262 -> 0.261 | No clear effect |
Surprising findings
- Removing the auxiliary load-balancing loss caused expert collapse into a smaller FFN and worse performance.
Limitations
- Gains are reported on six long-term forecasting benchmarks, which are a narrow evaluation of forecasting ability.
- Comparisons use a unified evaluation pipeline built by the authors rather than original reported numbers.
Experimental setup
Datasets: Time-300B ETTh1 ETTh2 ETTm1 ETTm2 Weather Global Temp
Benchmarks: ETTh1 ETTh2 ETTm1 ETTm2 Weather Global Temp
Metrics: MSE MAE
Baselines: Moirai TimesFM Moment Chronos iTransformer TimeMixer TimesNet PatchTST
Builds on: Switch Transformer DeepSeekMoE Moirai TimesFM
Closest papers by meaning
- Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts (ICML 2025 · similarity 0.89)
- Scaling Law for Time Series Forecasting (NeurIPS 2024 · similarity 0.89)
- TimePerceiver: An Encoder-Decoder Framework for Generalized Time-Series Forecasting (NeurIPS 2025 · similarity 0.89)
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.89)
- SynTSBench: Rethinking Temporal Pattern Learning in Deep Learning Models for Time Series (NeurIPS 2025 · similarity 0.89)
- Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.