Research map / Time series / Foundation models for time series
Learning to Embed Time Series Patches Independently
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PITS, a patch-independent self-supervised method for time series that pairs an unmasked-patch autoencoding task with a per-patch MLP encoder plus hierarchical complementary contrastive learning, matching or beating PatchTST with far fewer parameters.
Problem
Masked time series modeling pretrains Transformers to capture inter-patch dependencies, but such dependency learning may not be necessary or optimal for time series representation learning, and these models are costly.
Approach
Replace masked-patch prediction with patch-independent autoencoding of unmasked patches, use a per-patch MLP encoder instead of attention, add dropout to avoid identity mapping, and apply hierarchical contrastive learning on two complementary random masked views of the same series.
Key results
- Average forecasting MSE of 0.301 (PITS fine-tuned) vs 0.314 for PatchTST* in the self-supervised setting across seven datasets and four horizons.
- PITS uses 5,772 parameters vs 406,028 for PatchTST, with pretraining at 15-25 minutes vs 77 minutes and inference 3.3 s vs 7.5 s on ETTm2.
- PI reconstruction outperforms PD masked prediction across all four architectures in the ablation (e.g., MLP average gain +3.2%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PITS instead of self-supervised PatchTST | Multivariate time series forecasting · Seven forecasting datasets, average across four horizons | MSE: 0.314 -> 0.301 | Helped |
| Add complementary contrastive learning (ablation) | PITS · Multivariate time series forecasting | MSE: 0.304 -> 0.301 | Helped |
| Use hierarchical rather than non-hierarchical complementary CL (ablation) | PITS · ETTm2 forecasting | Avg. MSE: 0.252 -> 0.244 | Helped |
| Add hierarchical complementary CL (ablation) | PITS · ETTm2 forecasting | Pretrain time (min): 15 -> 25 | Hurt |
| Replace the PD task with patch-independent autoencoding (ablation) | Linear encoder · ETT forecasting, average across four datasets and four horizons | MSE: 0.342 -> 0.340 | Helped |
| Replace the PD task with patch-independent autoencoding (ablation) | MLP encoder · ETT forecasting, average across four datasets and four horizons | MSE: 0.348 -> 0.337 | Helped |
| Replace the PD task with patch-independent autoencoding (ablation) | MLP-Mixer encoder · ETT forecasting, average across four datasets and four horizons | MSE: 0.350 -> 0.339 | Helped |
| Replace the PD task with patch-independent autoencoding (ablation) | Transformer encoder · ETT forecasting, average across four datasets and four horizons | MSE: 0.351 -> 0.341 | Helped |
| Use PI rather than PD pretraining | Time-series forecasting models · Forecasting under distribution shifts | MSE difference | Helped |
Surprising findings
- Pretraining with the PD masked-prediction task performs worse than trivial zero-filled input baselines on ETT datasets.
- Simple per-patch MLP outperforms Transformer and MLP-Mixer encoders, contrary to the expectation that inter-patch attention is necessary.
Limitations
- Gains over PatchTST are modest on several datasets, and PatchTST remains competitive or better on some horizons (e.g., Weather, Electricity).
- Dropout is required to avoid trivial identity mapping when hidden dimension exceeds patch size.
Experimental setup
Datasets: ETTh1 ETTh2 ETTm1 ETTm2 Weather Traffic Electricity SleepEEG Epilepsy FD-B Gesture EMG
Benchmarks: ETTh1 ETTh2 ETTm1 ETTm2 Weather Traffic Electricity Epilepsy SleepEEG FD-B Gesture EMG
Metrics: MSE MAE accuracy precision recall F1
Baselines: PatchTST SimMTM FEDformer Autoformer DLinear TSMixer TST TimeMAE TS2Vec TF-C
Builds on: PatchTST SimMTM TS2Vec DLinear
Closest papers by meaning
- Multi-Patch Prediction: Adapting Language Models for Time Series Representation Learning (ICML 2024 · similarity 0.90)
- Frequency-Masked Embedding Inference: A Non-Contrastive Approach for Time Series Representation Learning (AAAI 2025 · similarity 0.88)
- Language Pre-training Guided Masking Representation Learning for Time Series Classification (AAAI 2025 · similarity 0.88)
- TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation (ICML 2025 · similarity 0.86)
- IMTS is Worth Time $\times$ Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction (ICML 2025 · similarity 0.86)
- T-Rep: Representation Learning for Time Series using Time-Embeddings (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.