Research map / Time series / Foundation models for time series
Towards Neural Scaling Laws for Time Series Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically studies neural scaling laws for time series foundation models on in-distribution and out-of-distribution data, comparing encoder-only vs decoder-only Transformers and the Moirai and Chronos architectures.
Problem
Prior TSFM scaling-law work covers only in-distribution data and does not examine how model architecture affects scalability, including OOD generalization.
Approach
Trains encoder-only and decoder-only Transformer TSFMs over 1K to 100M-1B parameters, compute budgets, and data sizes (10M to 1B points from a filtered LOTSA-based corpus), fits power laws for ID and OOD NLL and MAPE, and compares architectural designs of Moirai and Chronos-T5 for their scaling behavior.
Key results
- ID and OOD NLL show similar power-law scaling with parameter count and compute, with the same slope but a constant offset.
- Encoder-only Transformers show slightly better parameter scalability and ID performance than decoder-only Transformers, with similar OOD scalability.
- Moirai and Chronos-T5 improve ID forecasting over the baselines but their gains do not extend to OOD data; Moirai is overtaken by the encoder-only Transformer on OOD as size grows.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase parameter count | Encoder-only Transformer · ID forecasting | NLL | Helped |
| Increase parameter count | Encoder-only Transformer · OOD forecasting | NLL | Helped |
| Increase compute budget | Encoder-only Transformer · ID and OOD forecasting | NLL | Helped |
| Increase pre-training dataset size | Encoder-only Transformer · ID and OOD forecasting | NLL | Helped |
| Increase model size, compute, or dataset size | Encoder-only Transformer · OOD forecasting | MAPE | Helped |
| Use encoder-only instead of decoder-only architecture | Transformer TSFMs · ID forecasting | NLL | Helped |
| Use encoder-only instead of decoder-only architecture | Transformer TSFMs · OOD forecasting | NLL | No clear effect |
| Use Moirai architecture and design | Encoder-only Transformer · ID forecasting | NLL | Helped |
| Use Moirai architecture and design | Encoder-only Transformer · OOD forecasting | NLL | Hurt |
| Use Chronos-T5 architecture and design | Decoder-only Transformer · ID forecasting | SMAPE | Helped |
| Use Chronos-T5 architecture and design | Decoder-only Transformer · OOD forecasting | SMAPE | Hurt |
| Use Student-t mixture instead of Gaussian mixture (ablation) | Encoder-only Transformer · ID and OOD forecasting | NLL | Helped |
Surprising findings
- Chronos-T5 shows very small NLL power-law exponents, which the authors attribute to discrete probability prediction making NLL insensitive to distance.
- Some OOD datasets (us births, nn5 daily, australian electricity) show emergent-like jumps at around 10M parameters rather than smooth power-law behavior.
Limitations
- Only univariate forecasting is studied; multivariate scaling is left for future work.
- Context length and forecast horizon effects on scaling are not analyzed.
- Scaling factors are studied independently rather than jointly under a unified model.
Experimental setup
Datasets: LOTSA LSF (ETT, Electricity, Weather, Traffic) Monash
Benchmarks: LSF Monash
Metrics: NLL MAPE SMAPE MASE CRPS
Baselines: Moirai Chronos-T5 Exponential smoothing (ETS)
Builds on: neural scaling laws PatchTST Moirai Chronos Timer
Closest papers by meaning
- Multi-Scale Finetuning for Encoder-based Time Series Foundation Models (NeurIPS 2025 · similarity 0.90)
- Exploring Representations and Interventions in Time Series Foundation Models (ICML 2025 · similarity 0.90)
- Less is More: Unlocking Specialization of Time Series Foundation Models via Structured Pruning (NeurIPS 2025 · similarity 0.89)
- Revisiting the Generic Transformer: Deconstructing a Strong Baseline for Time Series Foundation Models (NeurIPS 2026 · similarity 0.88)
- On the Invariance and Generality of Neural Scaling Laws (NeurIPS 2026 · similarity 0.88)
- OATS: Online Data Augmentation for Time Series Foundation Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.