Research map / Time series / Foundation models for time series

Revisiting the Generic Transformer: Deconstructing a Strong Baseline for Time Series Foundation Models

NeurIPS 2026 · Time series › Foundation models for time series · approach: Time series foundation models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that a generic encoder-only patch Transformer with contiguous patch masking, mask-aware normalization and quantile outputs reaches SOTA zero-shot GIFT-Eval results when trained on a curated data mix, and ablates data, scaling, and training choices.

Problem

Time Series Foundation Models are trained under heterogeneous recipes (data, processing, training), making it unclear whether gains come from architecture or data engineering.

Approach

Uses a standard encoder-only patch Transformer with contiguous patch masking (masked autoencoding), mask-aware instance normalization, and a quantile head, trained on controlled open data (GIFT-Eval-Pretrain, KernelSynth, TSMixup), and systematically ablates scaling, data composition, and training techniques to isolate what drives performance.

Key results

• Pretrained variant achieves MASE 0.696 and CRPS 0.480 on GIFT-Eval (aggregated over 97 cases), the best across the compared TSFMs per the paper.
• Zero-shot variant (no test-distribution exposure) remains highly competitive with recent specialized TSFMs on GIFT-Eval.
• Contiguous patch masking with block size 8 improves over scattered masking (N_CPM=1): MASE 0.696 vs 0.750, CRPS 0.480 vs 0.514.

Surprising findings

• Excluding leaked test cases, the Pretrained model still outperforms other baselines on the remaining 74 cases, suggesting leakage does not fully explain gains.
• Deep-narrow models outperform shallow-wide models at similar parameter counts.

Limitations

• Univariate zero-shot forecasting only; multivariate and fine-tuning/adaptation not evaluated.
• Baseline results taken from the GIFT-Eval leaderboard, so training setups are heterogeneous and not controlled.
• Maximum reliable forecast length is about 2000 steps on a synthetic sine test; degradation at longer horizons.

Experimental setup

Datasets: GIFT-Eval-Pretrain KernelSynth TSMixup GIFT-Eval

Benchmarks: GIFT-Eval

Metrics: MASE CRPS Rank

Baselines: TabPFN-TS VisionTS Sundial Moirai-2 Toto YingLong FlowState TimesFM-2.5 Tirex Chronos-2 PatchTST

Builds on: PatchTST Reversible Instance Normalization TSMixup KernelSynth Contiguous Patch Masking

Closest papers by meaning

• Unlocking the Power of Patch: Patch-Based MLP for Long-Term Time Series Forecasting (AAAI 2025 · similarity 0.89)
• A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle (ICML 2025 · similarity 0.88)
• DeformableTST: Transformer for Time Series Forecasting without Over-reliance on Patching (NeurIPS 2024 · similarity 0.88)
• Towards Neural Scaling Laws for Time Series Foundation Models (ICLR 2025 · similarity 0.88)
• SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention (ICML 2024 · similarity 0.87)
• CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting (ICLR 2024 · similarity 0.87)