Research map / Time series / Foundation models for time series
Towards a General Time Series Forecasting Model with Unified Representation and Adaptive Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ROSE, a lightweight (7.4M parameter) general time series forecasting foundation model pre-trained on ~887M time points with decomposed frequency learning (multi-frequency masking reconstruction) and a Time Series Register that stores domain-specific embeddings for Top-K adaptive transfer.
Problem
General time series forecasting models pre-trained on multi-domain data struggle to learn unified representations across heterogeneous frequency content and to adaptively transfer domain-specific information to diverse downstream tasks, while existing foundation models rely mainly on scaling.
Approach
Replace time-domain masking with multi-threshold frequency-domain masking (high- or low-pass masks applied several times per series) for reconstruction-based pre-training, and add a learnable register of cluster-center vectors that stores domain-specific information; at fine-tuning, Top-K register vectors nearest to the input embedding are averaged and modulated by a learnable low-rank matrix.
Key results
- ROSE achieves full-shot average MSE reduction of about 15% over six full-data baselines on the seven benchmarks (e.g., ETTh1 MSE 0.391 vs 0.406 for S2IP-LLM).
- With only 10% fine-tuning data, ROSE beats the full-data baselines' average MSE by over 12%, per the paper.
- In zero-shot forecasting, ROSE reports about 15% average MSE reduction and a 7.4M parameter count, roughly one-tenth of Timer's.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ROSE instead of the strongest listed full-shot baseline | ROSE · Full-shot forecasting across seven benchmarks | MSE: average MSE reduction of 15% | Helped |
| Use ROSE instead of the strongest listed zero-shot baseline | ROSE · Zero-shot forecasting across benchmarks | MSE: average reduction of 15% in MSE | Helped |
| Add TS-Register (ablation) | ROSE · 10% few-shot ETTh1, ETTh2, ETTm1, and ETTm2 | MSE: 0.418 -> 0.397; 0.355 -> 0.335; 0.354 -> 0.349; 0.256 -> 0.250 | Helped |
| Add the prediction task (ablation) | ROSE · 10% few-shot ETTh1, ETTh2, ETTm1, and ETTm2 | MSE: 0.422 -> 0.397; 0.372 -> 0.335; 0.360 -> 0.349; 0.257 -> 0.250 | Helped |
| Add the reconstruction task (ablation) | ROSE · 10% few-shot ETTm1 and ETTm2 | MSE: 0.387 -> 0.349; 0.269 -> 0.250 | Helped |
| Replace multi-frequency masking with random frequency masking (ablation) | ROSE · 10% few-shot ETTm1 and ETTm2 | MSE: 0.349 -> 0.381; 0.250 -> 0.261 | Hurt |
| Replace multi-frequency masking with patch masking (ablation) | ROSE · 10% few-shot ETTm1 and ETTm2 | MSE: 0.349 -> 0.378; 0.250 -> 0.261 | Hurt |
| Replace multi-frequency masking with multi-patch masking (ablation) | ROSE · 10% few-shot ETT datasets | MSE: 0.349 -> 0.356; 0.250 -> 0.259; 0.397 -> 0.404; 0.335 -> 0.349 | Hurt |
| Increase ROSE model parameters from 2M to 4.5M (ablation) | ROSE · Few-shot forecasting with 5% and 10% downstream data | performance improvement: 10.37% and 9.34% improvements | Helped |
| Increase pre-training dataset size from 178M to 887M (ablation) | ROSE · Downstream forecasting | performance improvement: 7.4% and 4.8% respectively | Helped |
| Use 4 masked series instead of fewer series (ablation) | ROSE · 10% few-shot average across four ETT datasets | downstream performance | Mixed |
| Use 3 register tokens (ablation) | ROSE · 10% few-shot ETT datasets | performance | Helped |
Surprising findings
- Without the reconstruction task, the model shows negative transfer on ETTm1 and ETTm2 in the 10% few-shot setting, per the ablation.
- Random frequency masking and patch masking cause negative transfer on ETTm1 and ETTm2 relative to training from scratch.
Limitations
- Zero-shot results for Moment rely on a custom reconstruction-based forecasting procedure that Moment does not officially support.
- Evaluation is limited to seven forecasting benchmarks and is not extended to other tasks such as classification.
- Authors note their pre-training dataset is relatively modest (887M points) compared with other foundation models that use far more data.
Experimental setup
Datasets: Monash UEA UCR Pems03 Pems04 Pems07 Pems08 Pems-bay ETTh1 ETTh2 ETTm1 ETTm2
Benchmarks: ETTh1 ETTh2 ETTm1 ETTm2 Weather Electricity Traffic M4
Metrics: MSE MAE SMAPE MASE OWA
Baselines: iTransformer PatchTST TimesNet DLinear GPT4TS S2IP-LLM Timer MOIRAI Chronos TimesFM Moment
Builds on: PatchTST masked modeling LoRA
Closest papers by meaning
- Large Pre-trained time series models for cross-domain Time series analysis tasks (NeurIPS 2024 · similarity 0.90)
- TimePerceiver: An Encoder-Decoder Framework for Generalized Time-Series Forecasting (NeurIPS 2025 · similarity 0.90)
- Learning Pattern-Specific Experts for Time Series Forecasting Under Patch-level Distribution Shift (NeurIPS 2025 · similarity 0.89)
- TS-RAG: Retrieval-Augmented Generation based Time Series Foundation Models are Stronger Zero-Shot Forecaster (NeurIPS 2025 · similarity 0.89)
- Unified Training of Universal Time Series Forecasting Transformers (ICML 2024 · similarity 0.89)
- Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.