Research map / Time series / Foundation models for time series
TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TEST, a TS-for-LLM method that trains a causal TCN encoder with instance-wise, feature-wise, and text-prototype-aligned contrast so time series embeddings match a frozen LLM's word-embedding space, then uses learnable soft prompts for classification, forecasting, and representation tasks.
Problem
Time-series tasks are hard for pre-trained LLMs, which perform near random on TS classification and forecasting; fine-tuning LLMs for TS risks losing language ability and needs large TS data.
Approach
Keep the LLM frozen and train an external TS encoder whose embeddings are contrastively aligned to a set of text-token prototypes (cluster centers of the LLM embedding table), then feed the embeddings plus learned soft prompts directly into the LLM, skipping its embedding layer.
Key results
- On long-term forecasting with GPT2-Medium backbone, TEST achieves best average MSE on 5 of 8 datasets, e.g., ETTh1 avg MSE 0.414 vs GPT4TS 0.427.
- Using TEST, GPT2-774M improves classification accuracy by at least 18% on univariate and 25% on multivariate TS over original LLM QA baselines.
- Few-shot forecasting with 10% training data yields a relative average MSE reduction of 23.5% per the authors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TEST embeddings and soft prompts with GPT2-774M instead of the original LLM classification setup | GPT2-774M · UCR univariate and UEA multivariate classification | accuracy: at least 18% for univariate TS and 25% for multivariate TS | Helped |
| Choose text prototypes that better represent the full LLM text embedding space (ablation) | TEST · UCR and UEA classification | accuracy | Helped |
| Use a soft prompt instead of the hard classification prompt (ablation) | TEST · UCR and UEA classification | accuracy: at least 10% | Mixed |
| Initialize the prompt from the task description instead of randomly (ablation) | TEST · UCR and UEA classification | training convergence speed | Helped |
| Set prompt length to 10 (ablation) | TEST · UCR and UEA classification with a 1B model | accuracy | Helped |
| Use TEST for forecasting with 10% of training time steps | LLM-based forecasting models · Few-shot forecasting | MSE: 23.5% | Helped |
Surprising findings
- Original LLMs with QA-style prompting perform near random on TS classification, especially multivariate.
Limitations
- Performance comparable to SOTA only when LLM is ~300M+ parameters; small LLMs lag
- Text-prototype semantics are not human-interpretable
- Weak robustness noted in related-work comparison
Experimental setup
Datasets: UCR archive UEA archive ETTh ETTm Weather Electricity Traffic ILI TSER archive M3 M4 TOURISM
Benchmarks: UCR UEA ETTh1 ETTh2 ETTm1 ETTm2 Weather Electricity Traffic ILI TSER M3
Metrics: accuracy MSE MAE
Baselines: GPT4TS TimesNet DLinear Informer TS2Vec CoST T-Loss TCN DWT 1NNED PatchTST FEDformer
Builds on: p-tuning TS2Vec GPT4TS TimesNet
Closest papers by meaning
- Context-Alignment: Activating and Enhancing LLMs Capabilities in Time Series (ICLR 2025 · similarity 0.90)
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.89)
- $S^2$IP-LLM: Semantic Space Informed Prompt Learning with LLM for Time Series Forecasting (ICML 2024 · similarity 0.89)
- TimeCMA: Towards LLM-Empowered Multivariate Time Series Forecasting via Cross-Modality Alignment (AAAI 2025 · similarity 0.88)
- Multi-Patch Prediction: Adapting Language Models for Time Series Representation Learning (ICML 2024 · similarity 0.87)
- STEM-LTS: Integrating Semantic-Temporal Dynamics in LLM-driven Time Series Analysis (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.