Research map / Time series / Foundation models for time series

Context is Key: A Benchmark for Forecasting with Essential Textual Information

ICML 2025 · Time series › Foundation models for time series · approach: LLM-based time series forecasting · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the Context is Key (CiK) benchmark of 71 manually crafted forecasting tasks requiring essential textual context, a region-of-interest CRPS metric, and a simple Direct Prompt LLM forecaster that outperforms statistical, TSFM, and multimodal baselines.

Problem

Existing time series forecasting evaluations do not guarantee that textual context is useful, so it is unknown whether LLM-based and multimodal forecasters can truly integrate natural language context with numerical history.

Approach

Build a benchmark where every task is manually designed so accurate forecasts are only possible by using the textual context, score forecasts with a region-of-interest CRPS that adds constraint-violation penalties, and prompt LLMs to output full forecasts directly in structured form.

Key results

• Llama-3.1-405B-Instruct with Direct Prompt achieves the best average RCRPS of 0.159, and improves by 67.1% with context versus without.
• GPT-4o with Direct Prompt has the best average rank (4.381) but worse average RCRPS (0.274) due to significant failures on some tasks.
• Human annotators judged context useful for 94.7% of annotations; LLM critique (GPT-4o) judged context useful for all 71 tasks.

Measured effects reported in the paper

Surprising findings

• Instruction tuning generally degrades LLMP forecasting performance; Llama-3 models roughly halve in performance after instruction tuning.
• Without context, LLM-based forecasters do not dominate quantitative models, and GPT-4o and Llama-3.1-405B Direct Prompt perform much worse without context.

Limitations

• Benchmark limited to natural language context and excludes multivariate time series.
• Memorization contamination cannot be fully ruled out for publicly available data.
• Dataset-specific methods like ChatTime, UniTime and Time-LLM may perform better with training sets, which CiK does not provide.

Experimental setup

Datasets: PeMS traffic occupancy Solar irradiance and cloud cover (Americas 2022) Solar photovoltaic power (Alabama 2006) Montreal Fire Department intervention logs Causal Chambers wind tunnel FRED unemployment NN5 ATM withdrawals Monash electricity

Benchmarks: Context is Key (CiK)

Metrics: RCRPS average rank CRPS

Baselines: LLMP UniTime Time-LLM Lag-Llama Chronos-Large TimeGEN Moirai-Large ARIMA ETS Exponential Smoothing

Builds on: LLM Processes CRPS Chronos Time-LLM

Closest papers by meaning

• Rethinking Post-Training Recipes for Multimodal Time-Series Forecasting (NeurIPS 2026 · similarity 0.89)
• Approaching Human-Level Forecasting with Language Models (NeurIPS 2024 · similarity 0.89)
• AutoTimes: Autoregressive Time Series Forecasters via Large Language Models (NeurIPS 2024 · similarity 0.88)
• TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loop (NeurIPS 2025 · similarity 0.88)
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.88)
• TimeCAP: Learning to Contextualize, Augment, and Predict Time Series Events with Large Language Model Agents (AAAI 2025 · similarity 0.87)