Research map / Time series / Foundation models for time series
Context-Alignment: Activating and Enhancing LLMs Capabilities in Time Series
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Context-Alignment, a paradigm that aligns time series with language context via dual-scale GNNs (DSCA-GNNs) and a few-shot instantiation (FSCA) inserted into GPT-2 layers, reporting gains in long-term, few-shot and zero-shot forecasting.
Problem
Token-level alignment of time series with LLM embeddings fails to exploit LLMs' strength in linguistic logic and structure, limiting LLM performance on time series tasks (forecasting and classification).
Approach
Context-level alignment via dual-scale graph neural networks: a coarse-grained GNN treats each TS segment and prompt as a node with directed logical edges, a fine-grained GNN keeps token nodes, and their interaction is learned; FSCA adds demonstration examples in the prompt and the module is inserted at multiple GPT-2 layers.
Key results
- FSCA reports average long-term MSE 3.1% lower than PatchTST, and 7.3%, 12.2%, 16.6% lower than S2IP-LLM, Time-LLM, GPT4TS respectively.
- In 5%-data few-shot forecasting on ETT, FSCA average MSE 0.415 vs 0.445 for S2IP-LLM (about 6.7% reduction).
- In zero-shot cross-domain ETT forecasting, FSCA average MSE 0.357 vs 0.412 for PatchTST (about 13.3% improvement).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add context-alignment with FSCA | GPT-2 for long-term forecasting · ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity, Traffic, and ILI | MSE: 3.1% MSE reduction compared to the second-best method, PatchTST | Helped |
| Add FSCA few-shot demonstrations and context-alignment | GPT-2 for forecasting with 5% training data · ETT few-shot average | MSE: 0.445~ -> 0.415~ | Helped |
| Add FSCA context-alignment | GPT-2 for cross-domain forecasting · ETT zero-shot average | MSE: 0.412 -> 0.357 | Helped |
| Add DSCA-GNNs to VCA (ablation) | VCA on ETTh1 · ETTh1 | MSE: 0.435 -> 0.417 | Helped |
| Add Dual-Scale GNNs (ablation) | FSCA · ETTh1 long-term forecasting | MSE: 0.441 -> 0.394 | Helped |
| Add Dual-Scale GNNs (ablation) | FSCA · ETTm1 long-term forecasting | MSE: 0.379 -> 0.342 | Helped |
| Add Dual-Scale GNNs (ablation) | FSCA · FaceDetection classification | accuracy: 67.7 -> 70.4 | Helped |
| Use semantically guided rather than randomly initialized adjacency (ablation) | FSCA · ETTh1 long-term forecasting | MSE: 0.463 -> 0.441 | Helped |
| Add the coarse-grained branch (ablation) | FSCA · ETTh1 long-term forecasting | MSE: 0.401 -> 0.394 | Helped |
| Use 4 GPT-2 layers rather than 2 (ablation) | FSCA · ETTh1 long-term forecasting | MSE: 0.402 -> 0.396 | Helped |
| Use 6 GPT-2 layers rather than 4 (ablation) | FSCA · ETTh1 long-term forecasting | MSE: 0.396 -> 0.399 | Hurt |
| Increase GPT-2 depth to 8, 10, or 12 layers (ablation) | FSCA · ETTh1 long-term forecasting | MSE: 0.396 -> 0.418; 0.396 -> 0.439; 0.396 -> 0.455 | Hurt |
Surprising findings
- Random initialization of the adjacency matrix degrades performance relative to the dual-scale GNN without it, indicating incorrect logical structure harms the model.
Limitations
- Classification uses VCA instead of FSCA for multi-class datasets due to GPT-2 input length constraints.
- Performance depends on insertion position and number of LLM layers, which vary by task and require tuning.
Experimental setup
Datasets: ETTh1 ETTh2 ETTm1 ETTm2 Weather Electricity Traffic ILI M4 UEA classification datasets
Benchmarks: ETTh1 ETTh2 ETTm1 ETTm2 Weather Electricity Traffic ILI M4 UEA
Metrics: MSE MAE SMAPE MASE OWA accuracy
Baselines: GPT4TS Time-LLM S2IP-LLM iTransformer PatchTST DLinear TimesNet FEDformer Stationary ETSformer N-HiTS N-BEATS
Builds on: GPT4TS Time-LLM S2IP-LLM
Closest papers by meaning
- TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series (ICLR 2024 · similarity 0.90)
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.89)
- TimeCMA: Towards LLM-Empowered Multivariate Time Series Forecasting via Cross-Modality Alignment (AAAI 2025 · similarity 0.89)
- Long-Short Alignment for Effective Long-Context Modeling in LLMs (ICML 2025 · similarity 0.89)
- STEM-LTS: Integrating Semantic-Temporal Dynamics in LLM-driven Time Series Analysis (AAAI 2025 · similarity 0.88)
- In-context Time Series Predictor (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.