Research map / Time series / Foundation models for time series
Are Language Models Actually Useful for Time Series Forecasting?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that LLM-based time series forecasters (Time-LLM, OneFitsAll/GPT4TS, CALF) perform no better than ablations that remove the LLM or replace it with a simple attention/transformer layer, while costing far more compute.
Problem
Whether pretrained language models actually contribute to time series forecasting in recent LLM-based forecasters, given their large computational cost.
Approach
Three ablations per LLM-based forecaster (removing the LLM, replacing it with one attention layer, or with one transformer block), plus pretraining-vs-random-init, input-shuffling, few-shot, and encoder studies, showing the LLM component is not what drives forecasting performance.
Key results
- Ablations without the LLM outperform Time-LLM in 35/40 cases, CALF in 31/40, and OneFitsAll in 29/40 (MAE, averaged over horizons).
- Time-LLM has about 6642M parameters and takes 3003 minutes to train on Weather, versus about 0.245M parameters and roughly 2 minutes for the ablations.
- Randomly initialized GPT-2 in CALF (woPre+FT) wins 8 of 8 dataset comparisons on MAE and MSE vs 3 for pretrained-plus-finetune.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove the LLM from Time-LLM (ablation) | Time-LLM · 13 datasets, averaged prediction lengths | forecasting wins: 0 -> 12 wins | Helped |
| Replace the LLM with a randomly initialized transformer block in Time-LLM (ablation) | Time-LLM · 13 datasets, averaged prediction lengths | forecasting wins: 0 -> 12 wins | Helped |
| Replace the LLM with a randomly initialized attention layer in Time-LLM (ablation) | Time-LLM · 13 datasets, averaged prediction lengths | forecasting wins: 0 -> 2 wins | Helped |
| Remove the LLM from CALF (ablation) | CALF · 13 datasets, averaged prediction lengths | forecasting wins: 4 -> 7 wins | Helped |
| Replace the LLM with a randomly initialized transformer block in CALF (ablation) | CALF · 13 datasets, averaged prediction lengths | forecasting wins: 4 -> 11 wins | Helped |
| Replace the LLM with a randomly initialized attention layer in CALF (ablation) | CALF · 13 datasets, averaged prediction lengths | forecasting wins: 4 -> 4 wins | No clear effect |
| Remove the LLM from OneFitsAll (ablation) | OneFitsAll · 13 datasets, averaged prediction lengths | forecasting wins: 7 -> 11 wins | Helped |
| Replace the LLM with a randomly initialized attention layer in OneFitsAll (ablation) | OneFitsAll · 13 datasets, averaged prediction lengths | forecasting wins: 7 -> 3 wins | Hurt |
| Replace the LLM with a randomly initialized transformer block in OneFitsAll (ablation) | OneFitsAll · 13 datasets, averaged prediction lengths | forecasting wins: 7 -> 5 wins | Hurt |
| Randomly initialize GPT-2 weights before finetuning instead of using pretrained weights (ablation) | CALF · 8 datasets | forecasting wins: 3 -> 8 wins | Helped |
| Use a pretrained LLM without finetuning instead of pretrained LLM finetuning (ablation) | CALF · 8 datasets | forecasting wins: 3 -> 5 wins | Mixed |
| Shuffle the input time series (ablation) | Time-LLM and its ablations · ETTh1 and Illness | forecasting performance degradation | No clear effect |
Surprising findings
- Removing the LLM entirely or using a random-init LLM matches or beats the pretrained LLM-based forecasters.
- Shuffling input time series degrades LLM-based forecasters no more than their LLM-free ablations, suggesting no special sequence modeling from the LLM.
Limitations
- Evaluated only on forecasting, not classification or question answering.
- Only uniformly sampled time series; non-uniform sequences not studied.
- Only three LLM-based forecasters, all using GPT-2 or LLaMA backbones.
Experimental setup
Datasets: ETTh1 ETTh2 ETTm1 ETTm2 Illness Weather Traffic Electricity Exchange Rate Covid Deaths Taxi (30 min) NN5 (Daily)
Benchmarks: ETTh1 ETTh2 ETTm1 ETTm2 Illness Weather Traffic Electricity Exchange Rate Covid Deaths Taxi (30 min) NN5 (Daily)
Metrics: MAE MSE training time inference time parameter count
Baselines: Time-LLM OneFitsAll CALF PatchTST DLinear
Builds on: Time-LLM OneFitsAll CALF PatchTST DLinear
Closest papers by meaning
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.92)
- Position: What Can Large Language Models Tell Us about Time Series Analysis (ICML 2024 · similarity 0.91)
- AutoTimes: Autoregressive Time Series Forecasters via Large Language Models (NeurIPS 2024 · similarity 0.91)
- Can LLMs Understand Time Series Anomalies? (ICLR 2025 · similarity 0.90)
- In-context Time Series Predictor (ICLR 2025 · similarity 0.90)
- TimeCAP: Learning to Contextualize, Augment, and Predict Time Series Events with Large Language Model Agents (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.