Research map / Time series / Foundation models for time series
$S^2$IP-LLM: Semantic Space Informed Prompt Learning with LLM for Time Series Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S²IP-LLM, which aligns decomposed-and-patched time series embeddings with semantic anchors derived from GPT-2 word token embeddings via cosine similarity, retrieves top-K anchors as prefix prompts, and fine-tunes GPT-2 layer norms and positional embeddings for forecasting.
Problem
Time series forecasting with pre-trained LLMs is limited because the LLM semantic space is underexplored and patch tokenization may not capture subtle component variations; dataset text descriptions are not always available.
Approach
Derive a small set of semantic anchors from pre-trained word token embeddings via a mapping function, align them with decomposed time series patch embeddings by cosine-similarity maximization in a joint space, and use the top-K retrieved anchors as prefix prompts for a frozen GPT-2 backbone.
Key results
- Long-term forecasting: S²IP-LLM achieves best average MSE on Weather (0.222), Electricity (0.161), ETTh1 (0.406), ETTh2 (0.347), ETTm1 (0.343), and ETTm2 (0.257) versus Time-LLM(G) and OFA among LLM baselines.
- Short-term M4 average: SMAPE 12.021, MASE 1.612, OWA 0.857, better than Time-LLM(G) (OWA 0.913) and OFA (OWA 0.94), and slightly better than PatchTST (OWA 0.869).
- Few-shot (10% training data): S²IP-LLM is best or second-best on most datasets, e.g., Weather MSE 0.233 vs Time-LLM(G) 0.237.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add semantic-space alignment and prompting (ablation) | S²IP-LLM · ETTh2 and ETTm2 | forecasting performance | Helped |
| Add decomposition (ablation) | S²IP-LLM · ETTh2 and ETTm2 | forecasting performance | Helped |
| Increase prompt length within 2 to 8 (ablation) | S²IP-LLM · ETTh2 and ETTm2 | forecasting performance | Helped |
| Increase prompt length to 16 or 32 (ablation) | S²IP-LLM · ETTh2 and ETTm2 | forecasting performance | Hurt |
| Increase alignment weight λ from 0 to 0.05 (ablation) | S²IP-LLM · ETTh2 and ETTm2 | forecasting performance | Helped |
| Use an excessively large alignment weight λ (ablation) | S²IP-LLM · ETTh2 and ETTm2 | forecasting performance | Hurt |
| Increase the number of semantic anchors (ablation) | S²IP-LLM · ETTh2 and ETTm2 | forecasting performance | Helped |
| Use semantic-space-informed prompting rather than the baseline method | S²IP-LLM · M4 (weighted average) | OWA: 0.869 -> 0.857 | Helped |
Surprising findings
- Patching-based methods (OFA, PatchTST, Time-LLM) treat multivariate series as independent univariate series, which the authors suggest gives them more training data and may explain their advantage over multivariate transformers.
Limitations
- Gains over strongest baselines are often small and the best result is not uniform across datasets (e.g., Traffic where iTransformer or PatchTST is better).
- Comparisons with transformer baselines may be unfair because patching-based methods treat variables as independent univariate series.
Experimental setup
Datasets: ETTh1 ETTh2 ETTm1 ETTm2 Weather Electricity Traffic M4
Benchmarks: ETTh1 ETTh2 ETTm1 ETTm2 Weather Electricity Traffic M4
Metrics: MSE MAE SMAPE MASE OWA
Baselines: Time-LLM OFA iTransformer DLinear PatchTST TimesNet FEDformer Autoformer Non-stationary Transformer ETSformer Informer LightTS
Builds on: Time-LLM OFA PatchTST reversible instance normalization soft prompts
Closest papers by meaning
- TimeCMA: Towards LLM-Empowered Multivariate Time Series Forecasting via Cross-Modality Alignment (AAAI 2025 · similarity 0.93)
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.91)
- STEM-LTS: Integrating Semantic-Temporal Dynamics in LLM-driven Time Series Analysis (AAAI 2025 · similarity 0.89)
- AutoTimes: Autoregressive Time Series Forecasters via Large Language Models (NeurIPS 2024 · similarity 0.89)
- TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series (ICLR 2024 · similarity 0.89)
- Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.