Research map / Time series / Foundation models for time series
Time-VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Time-VLM, which feeds time series (as images, text prompts and retrieved memory features) through a frozen pre-trained vision-language model and fuses the embeddings with temporal features for forecasting, reporting strong few-shot and zero-shot results.
Problem
Time series forecasting struggles to generalize in few-shot and zero-shot settings; existing text- or vision-augmented methods use only one modality and lack either fine-grained temporal detail or semantic context.
Approach
Combine a retrieval-augmented temporal learner, a vision-augmented learner that renders time series as three-channel images (FFT and periodicity encodings), and a text-augmented learner that generates statistical prompts, feeding both to a frozen VLM (ViLT by default) and fusing via cross-modal attention and gated fusion.
Key results
- On ETTh1 with 5% training data, Time-VLM reports MSE 0.442 vs 0.627 for Time-LLM (about 29.5% reduction).
- On M4 short-term forecasting, Time-VLM achieves SMAPE 11.894 and OWA 0.855, slightly better than Time-LLM (11.983 and 0.859).
- Time-VLM has 143.6M parameters versus Time-LLM's 3404.6M, and its memory use on ETTh1 is 2630 MiB versus 37723 MiB for Time-LLM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Time-VLM instead of Time-LLM | long-term forecasting · ETTh1 | MSE: 0.408 -> 0.405 | Helped |
| Use Time-VLM instead of Time-LLM | long-term forecasting · ETTm2 | MSE: 0.251 -> 0.248 | Helped |
| Use Time-VLM instead of Time-LLM | long-term forecasting · Electricity | MSE: 0.158 -> 0.172 | Hurt |
| Use Time-VLM instead of Time-LLM | long-term forecasting · Traffic | MSE: 0.388 -> 0.419 | Hurt |
| Use Time-VLM instead of Time-LLM | few-shot forecasting with 5% training data · ETTh1 | MSE: 0.627 -> 0.442 | Helped |
| Use Time-VLM instead of Time-LLM | few-shot forecasting with 5% training data · Weather | MSE: 0.260 -> 0.240 | Helped |
| Use Time-VLM instead of the strongest baseline | few-shot forecasting with 5% training data · ECL | MSE: 0.178 -> 0.218 | Hurt |
| Use Time-VLM instead of the strongest baseline | few-shot forecasting with 5% training data · Traffic | MSE: 0.418 -> 0.558 | Hurt |
| Use Time-VLM instead of Time-LLM | M4 short-term forecasting · M4 | SMAPE: 11.983 -> 11.894 | Helped |
| Add RAL (ablation) | Time-VLM · Weather (average over horizons) | MSE: 0.316 -> 0.233; 35.6% | Helped |
| Add RAL local memory branch (ablation) | Time-VLM · Weather (average over horizons) | MSE: 0.273 -> 0.233; 17.2% | Helped |
| Add RAL global memory branch (ablation) | Time-VLM · Weather (average over horizons) | MSE: 0.243 -> 0.233; 4.3% | Helped |
Surprising findings
- Removing the text-augmented learner causes only a 2.1% MSE increase, because ViLT yields very few textual tokens (11 of 156).
- Larger VLMs do not improve accuracy: BLIP-2 (3.7B) underperforms ViLT and CLIP on ETTh2.
Limitations
- The text-augmented component has limited impact because current VLMs have weak understanding of time series semantics.
- Full-shot performance lags specialized unimodal models on some datasets such as ECL and Traffic.
- Deployment on resource-constrained devices remains challenging.
Experimental setup
Datasets: ETTh1 ETTh2 ETTm1 ETTm2 Weather Electricity Traffic M4 MSCOCO
Benchmarks: ETTh1 ETTh2 ETTm1 ETTm2 Weather Electricity Traffic M4
Metrics: MSE MAE SMAPE MASE OWA
Baselines: Time-LLM GPT4TS LLMTime TimesNet PatchTST DLinear FEDformer Autoformer Stationary ETSformer LightTS Informer
Builds on: Time-LLM TimesNet PatchTST ViLT
Closest papers by meaning
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.90)
- ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data (AAAI 2025 · similarity 0.90)
- TimeCMA: Towards LLM-Empowered Multivariate Time Series Forecasting via Cross-Modality Alignment (AAAI 2025 · similarity 0.89)
- Rethinking Post-Training Recipes for Multimodal Time-Series Forecasting (NeurIPS 2026 · similarity 0.89)
- Empowering Time Series Analysis with Large-Scale Multimodal Pretraining (NeurIPS 2026 · similarity 0.89)
- TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loop (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.