Research map / Time series / Foundation models for time series
Multi-Patch Prediction: Adapting Language Models for Time Series Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts GPT-2 for time series via two-stage training (causal next-patch continual pre-training, then multi-patch prediction fine-tuning with a shared patch-wise linear decoder) and evaluates on forecasting, few-shot, and anomaly detection.
Problem
Adapting pretrained LLMs to produce general-purpose time-series representations; existing contrastive and mask-and-reconstruction methods misalign with forecasting and LLM pretraining, and sequence-level decoders overfit.
Approach
Reframe time-series forecasting as self-supervised multi-patch prediction: first continually pretrain a causal LLM with next-patch prediction, then fine-tune only layer norms and position embeddings for multi-patch prediction, decoding each patch independently with a shared patch-level linear layer instead of a sequence-level head.
Key results
- Long-term forecasting: average MSE improvement of 9.71% over GPT4TS and 2.03% reduction vs PatchTST.
- Few-shot forecasting (5% data) on ETT: average MSE 0.425 vs GPT4TS 0.465, an 8.6% reduction.
- Anomaly detection: average F1 of 87.51% across five datasets, SOTA versus GPT4TS 86.72%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add causal next-patch continual pre-training (stage 1) (ablation) | aLLM4TS · ETTh1 ablation | MSE: 0.413 -> 0.460 when removed | Helped |
| Use patch-wise decoder instead of sequence-level decoder (ablation) | aLLM4TS · ETTh1 ablation | MSE: 0.413 -> 0.455 when sequence-level decoder used | Helped |
| Add position-aware attention mask in stage 2 (ablation) | aLLM4TS · ETTh1 ablation | MSE: 0.413 -> 0.443 when removed | Helped |
| Use GPT-2 pretrained weights instead of random init (ablation) | aLLM4TS · ETTh1 ablation (6 layers) | MSE: 0.413 -> 0.437 when random init | Helped |
| Use 6 GPT-2 layers instead of 3 or 9/12 (ablation) | aLLM4TS · ETTh1 ablation | MSE: 3 layers 0.437, 9 layers 0.452, 12 layers 0.580 vs 6 layers 0.413 | Mixed |
| Replace masking-based pre-training with next-patch pre-training (Sta1 vs Masking+Sta2) (ablation) | aLLM4TS representation learning · ETTh1 representation learning | MSE: Average decline over 13.5% for masking variants | Helped |
| Adopt aLLM4TS vs GPT4TS on long-term forecasting (average over horizons) | aLLM4TS · Long-term forecasting | MSE: average 9.71% improvement over GPT4TS | Helped |
| Patch-wise decoder with sequence-level baseline comparison on M4 | aLLM4TS · M4 short-term forecasting | SMAPE: 11.950 vs TimesNet 11.829 | Mixed |
| Use aLLM4TS pipeline in few-shot forecasting (5% data) | aLLM4TS · Few-shot ETT 5% | MSE: 0.425 vs GPT4TS 0.465 | Helped |
Surprising findings
- Sequence-level decoder ablation (Masking+SD) shows over 13.4% deterioration, indicating overfitting of large sequence-wise heads.
Limitations
- Short-term forecasting on M4 is competitive with but does not beat TimesNet.
- Stage 1 pre-training uses a fixed set of datasets and relies on GPT-2 backbone choice; evaluated mainly on GPT-2 with first 6 layers.
Experimental setup
Datasets: Weather Traffic Electricity ILI ETTh1 ETTh2 ETTm1 ETTm2 M4 SMD MSL SMAP
Benchmarks: ETTh1 ETTh2 ILI Weather Traffic Electricity M4 ETT few-shot (5%) SMD MSL SMAP SWaT
Metrics: MSE MAE SMAPE F1-score
Baselines: GPT4TS Time-LLM PatchTST DLinear TimesNet FEDformer Autoformer N-HiTS N-BEATS TS2Vec
Builds on: GPT4TS PatchTST Time-LLM
Closest papers by meaning
- Learning to Embed Time Series Patches Independently (ICLR 2024 · similarity 0.90)
- TimeCMA: Towards LLM-Empowered Multivariate Time Series Forecasting via Cross-Modality Alignment (AAAI 2025 · similarity 0.89)
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.89)
- Unlocking the Power of Patch: Patch-Based MLP for Long-Term Time Series Forecasting (AAAI 2025 · similarity 0.88)
- Towards a General Time Series Forecasting Model with Unified Representation and Adaptive Transfer (ICML 2025 · similarity 0.88)
- $S^2$IP-LLM: Semantic Space Informed Prompt Learning with LLM for Time Series Forecasting (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.