Research map / Time series / Foundation models for time series
TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TimeSiam, a Siamese-network self-supervised pre-training framework that reconstructs masked current time-series subseries from randomly sampled past subseries, using learnable lineage embeddings to encode temporal distance.
Problem
Time series pre-training methods (masked modeling and contrastive learning) either distort temporal correlations through random masking or rely on instance-level similarity and fragile augmentations, failing to model temporal correlations.
Approach
Sample pairs of past and current subseries from the same series at different timestamps, process them with a shared Siamese encoder (PatchTST or iTransformer), add learnable lineage embeddings indexed by temporal distance, and train a cross-attention decoder to reconstruct the masked current subseries from the past one.
Key results
- In-domain PatchTST forecasting: TimeSiam average MSE 0.429 on ETTh1 vs 0.473 random init and 0.440 for the best baseline SimMTM; average MSE reduction of 5.7% across benchmarks.
- Cross-domain pre-training on TSLD-1G with TimeSiam_Large reaches 0.433 MSE on Traffic (+11.6% over random init).
- In-domain classification accuracy: TimeSiam 89.93% AD, 90.67% TDBrain, 91.32% PTB, versus best baseline SimMTM 86.19%, 84.81%, 90.04%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use past-to-current Siamese sampling rather than self-reconstruction (ablation) | TimeSiam · Traffic | MSE: 0.462 -> 0.453 | Helped |
| Increase the maximum sampling distance from 1 to 6 (ablation) | TimeSiam · Traffic | MSE: 0.454 -> 0.453 | Helped |
| Add lineage embeddings (ablation) | TimeSiam · ETTh1, ETTm1, Weather, Traffic, Exchange | MSE: 0.433 -> 0.429; 0.378 -> 0.374; 0.256 -> 0.252; 0.457 -> 0.453; 0.365 -> 0.353 | Helped |
| Use continuous channel-wise masking rather than binomial, continuous, mask-last, or channel-wise binomial masking (ablation) | TimeSiam · Traffic | MSE: 0.459 -> 0.453; 0.461 -> 0.453; 0.457 -> 0.453 | Helped |
| Use a 25% masking ratio rather than no masking, 15%, 50%, or 75% (ablation) | TimeSiam · Traffic | MSE: 0.460 -> 0.453; 0.459 -> 0.453; 0.455 -> 0.453; 0.457 -> 0.453 | Helped |
| Pre-train with TimeSiam instead of random initialization | PatchTST · In-domain forecasting benchmarks | MSE: +5.7% | Helped |
| Pre-train with TimeSiam instead of random initialization | iTransformer · In-domain forecasting benchmarks | MSE: 2.5% | Helped |
| Pre-train with TimeSiam rather than random initialization | TimeSiam · In-domain classification benchmarks | accuracy: 11.5% | Helped |
| Increase pre-training dataset size from TSLD-500M to TSLD-1G | TimeSiam Base · Traffic | MSE: 0.462 -> 0.454 | Helped |
| Increase model size from Base to Large | TimeSiam pre-trained on TSLD-1G · Traffic | MSE: 0.454 -> 0.433 | Helped |
| Use larger, more diverse cross-domain pre-training data rather than in-domain pre-training | TimeSiam · Classification: AD, TDBrain, PTB | accuracy: 90.47 vs. 89.93; 86.26 vs. 90.67; 90.45 vs. 91.32 | Mixed |
Surprising findings
- Cross-domain pre-training on the larger, more diverse TSLD data does not consistently beat in-domain pre-training for classification.
- Linear probing with only the projector trained gives forecasting MSE (0.365) on ETT close to full fine-tuning and better than random init (0.383).
Limitations
- Cross-domain transfer gains are not uniformly better than in-domain pre-training, especially for classification.
- Some baselines were not evaluated with iTransformer because their designs do not apply to it; Ti-MAE uses an unofficial implementation.
Experimental setup
Datasets: ETTh1 ETTh2 ETTm1 ETTm2 Weather Electricity Traffic Exchange AD TDBrain PTB TSLD-500M
Benchmarks: ETTh1 ETTh2 ETTm1 ETTm2 Weather Exchange Electricity Traffic AD TDBrain PTB
Metrics: MSE MAE accuracy precision recall F1 AUROC AUPRC
Baselines: CPC TNC TS2Vec CoST LaST TF-C COMET TST Ti-MAE PatchTST SimMTM
Builds on: PatchTST iTransformer SimMTM TS2Vec
Closest papers by meaning
- TStruct: Learning Shared Temporal Structures for Long-Term Time-Series Forecasting (NeurIPS 2026 · similarity 0.87)
- Language Pre-training Guided Masking Representation Learning for Time Series Classification (AAAI 2025 · similarity 0.86)
- TimesURL: Self-Supervised Contrastive Learning for Universal Time Series Representation Learning (AAAI 2024 · similarity 0.86)
- TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation (ICML 2025 · similarity 0.86)
- Frequency-Masked Embedding Inference: A Non-Contrastive Approach for Time Series Representation Learning (AAAI 2025 · similarity 0.86)
- TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.