Research map / Time series / Foundation models for time series
UP2ME: Univariate Pre-training to Multivariate Fine-tuning as a General-purpose Framework for Multivariate Time Series Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UP2ME, which pre-trains a masked autoencoder on univariate, channel-decoupled, variable-length multivariate time series, then fine-tunes with a sparse channel dependency graph for forecasting, imputation and anomaly detection.
Problem
Self-supervised pre-training for multivariate time series lags behind task-specific methods, and existing pre-trained models cannot run downstream tasks without parameter or architecture modification.
Approach
Pre-train an MAE on univariate instances (variable window length and channel decoupling) so temporal dependency is learned without cross-channel dependency; formulate forecasting, imputation and anomaly detection as mask-reconstruction for frozen immediate-reaction use; then fine-tune with Temporal-Channel layers over a sparse graph built from the frozen encoder's latent similarities.
Key results
- Fine-tuned UP2ME(FT) achieves the best forecasting mSE on all eight-dataset forecasting comparisons reported in Table 1 (e.g., ETTm1 0.341 vs 0.350 for TS2Vec-era best baseline per table).
- Frozen UP2ME(IR) is comparable to SOTA on ETTm1, Electricity, Traffic and GECCO forecasting without any parameter updates.
- UP2ME(FT) outperforms all other methods on 7 of 8 datasets in imputation, and on anomaly detection it approaches task-specific methods (e.g., SMD F1 83.31 vs DCdetector 84.40).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UP2ME(FT) instead of the strongest listed baseline | UP2ME · Forecasting on ETTm1, Weather, Electricity, Traffic, SMD, PSM, SWaT and GECCO | mSE: ETTm1: 0.350 -> 0.341; Weather: 0.228 -> 0.221; Electricity: 0.164 -> 0.155; Traffic: 0.395 -> 0.390; SMD: 0.893 -> 0.87 | Helped |
| Use UP2ME(FT) instead of the strongest listed baseline | UP2ME · Imputation on ETTm1, Weather, Electricity, Traffic, SMD, PSM and SWaT | mSE: ETTm1: 0.172 -> 0.128; Weather: 0.103 -> 0.079; Electricity: 0.140 -> 0.097; Traffic: 0.508 -> 0.294; SMD: 0.842 -> 0.75 | Helped |
| Use UP2ME(FT) instead of OCSVM | UP2ME · Anomaly detection on GECCO | F1: 52.41 -> 63.39 | Helped |
| Use UP2ME(FT) instead of the best task-specific detector | UP2ME · Anomaly detection on SMD, PSM and SWaT | F1: SMD: 90.98 -> 83.31; PSM: 97.50 -> 97.16; SWaT: 96.52 -> 93.85 | Hurt |
| Add variable window length during pre-training (ablation) | UP2ME · Forecasting on ETTm1 | mSE | Helped |
| Add channel decoupling during pre-training (ablation) | UP2ME · Forecasting on ETTm1 | mSE | Helped |
| Remove pre-training and train from scratch (ablation) | UP2ME · Forecasting on ETTm1 | mSE | Hurt |
| Set pre-training mask ratio to ≤30% (ablation) | UP2ME · Forecasting on ETTm1 | mSE | Hurt |
| Set pre-training mask ratio to ≥70% (ablation) | UP2ME · Forecasting on ETTm1 | mSE | Hurt |
| Replace the encoder-constructed graph with channel independence (ablation) | UP2ME(FT) · Imputation on Weather | mSE | Hurt |
| Construct the fine-tuning graph with the pre-trained encoder rather than random, Pearson or Euclidean methods (ablation) | UP2ME(FT) · Imputation on Weather | mSE | Helped |
| Increase graph hyper-parameter r (ablation) | UP2ME(FT) · Imputation on Weather | mSE | Mixed |
Surprising findings
- Frozen immediate-reaction mode without any fine-tuning already matches several task-specific methods on some datasets.
- On data-limited ETTm2 (<=10% data), the IR mode without fine-tuning performs best among compared methods.
Limitations
- Pre-trains one model per dataset; does not support multi-dataset or zero-shot generalization.
- Does not support classification in IR mode.
- Anomaly detection in fine-tuning only approaches, not surpasses, task-specific methods on the first three datasets.
Experimental setup
Datasets: ETTm1 Weather Electricity Traffic SMD PSM SWaT GECCO ETTm2
Benchmarks: ETTm1 Weather Electricity Traffic SMD PSM SWaT GECCO
Metrics: mSE mAE F1-score AP
Baselines: PatchTST DLinear Crossformer FEDformer TimesNet TS2Vec SimMTM SAITS GRIN DCdetector AnomalyTrans
Builds on: masked autoencoder PatchTST RevIN Graph Transformer
Closest papers by meaning
- Large Pre-trained time series models for cross-domain Time series analysis tasks (NeurIPS 2024 · similarity 0.87)
- Revitalizing Multivariate Time Series Forecasting: Learnable Decomposition with Inter-Series Dependencies and Intra-Series Variations Modeling (ICML 2024 · similarity 0.87)
- TiRex-2: Generalizing TiRex to Multivariate Data and Streaming (NeurIPS 2026 · similarity 0.86)
- AdaPTS: Adapting Univariate Foundation Models to Probabilistic Multivariate Time Series Forecasting (ICML 2025 · similarity 0.86)
- Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models (NeurIPS 2024 · similarity 0.86)
- Empowering Time Series Analysis with Large-Scale Multimodal Pretraining (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.