Research map / Time series / Foundation models for time series
Time-IMM: A Dataset and Benchmark for Irregular Multimodal Multivariate Time Series
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Time-IMM, a nine-dataset benchmark of cause-driven irregular multimodal (numeric + text) time series, and IMM-TSF, a modular forecasting library with timestamp-to-text and multimodality fusion modules showing text improves forecasting.
Problem
Real-world time series are irregular, multimodal, and messy, but existing benchmarks assume clean, regularly sampled, unimodal data and do not categorize causes of irregularity or support asynchronous text.
Approach
A taxonomy of nine irregularity causes (trigger-based, constraint-based, artifact-based) realized as nine datasets with asynchronous LLM-summarized text, plus a plug-and-play library with recency-weighted and Time2Vec cross-attention timestamp-to-text fusion and gated/cross-attention multimodality fusion.
Key results
- Best multimodal configurations reduce MSE by 6.71% on average across datasets, with gains up to 38.38% on datasets with informative text.
- t-PatchGNN is the strongest irregular model in the unimodal setting; TimeLLM and TTM remain competitive in the multimodal setting.
- Gated residual addition (GR-Add) consistently outperforms cross-attention addition (XAttn-Add); T2V-XAttn and RecAvg perform comparably; the choice of text encoder (GPT-2, BERT, Llama-3.1-8B, DeepSeek-7B) has no significant effect.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add textual signals to irregular time series forecasting | Time-IMM forecasting models · Time-IMM (average across datasets) | MSE: 6.71% | Helped |
| Add textual signals to irregular time series forecasting | Time-IMM forecasting models · Datasets with highly informative text | MSE: up to 38.38% | Helped |
| Use RecAvg instead of T2V-XAttn for timestamp-to-text fusion (ablation) | IMM-TSF · Time-IMM forecasting datasets | MSE | No clear effect |
| Use GR-Add instead of XAttn-Add for multimodality fusion (ablation) | IMM-TSF · Time-IMM forecasting datasets | MSE | Helped |
| Change the frozen text encoder among GPT-2, BERT, Llama 3.1 (8B), and DeepSeek (7B) (ablation) | IMM-TSF · Representative Time-IMM datasets | Forecasting performance | No clear effect |
| Add textual signals to forecasting on ClusterTrace | Time-IMM forecasting models · ClusterTrace | MSE | Helped |
| Add textual news to forecasting on FNSPID | Time-IMM forecasting models · FNSPID | MSE | No clear effect |
Surprising findings
- Increasing LLM text-encoder capacity (GPT-2 to Llama-3.1-8B or DeepSeek-7B) yields no clear benefit on irregular forecasting.
- Time2Vec-augmented cross-attention gives no notable gain over simple recency-weighted averaging.
Limitations
- Only English textual data; text is LLM-summarized and frozen encoders are not fine-tuned.
- Benchmark limited to forecasting with known query timestamps.
- Small dataset sizes (about 1-30 entities per dataset) relative to LLM pretraining scale.
Experimental setup
Datasets: GDELT RepoHealth MIMIC-IV FNSPID ClusterTrace (Alibaba GPU v2020) StudentLife ILINet CESNET EPA-Air
Benchmarks: Time-IMM
Metrics: MSE
Baselines: Informer DLinear PatchTST TimesNet TimeMixer TimeLLM TTM CRU Latent-ODE Neural Flow t-PatchGNN
Builds on: t-PatchGNN Time-MMD Time2Vec
Closest papers by meaning
- Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysis (NeurIPS 2024 · similarity 0.93)
- Empowering Time Series Analysis with Large-Scale Multimodal Pretraining (NeurIPS 2026 · similarity 0.90)
- Rethinking Post-Training Recipes for Multimodal Time-Series Forecasting (NeurIPS 2026 · similarity 0.88)
- Physiome-ODE: A Benchmark for Irregularly Sampled Multivariate Time-Series Forecasting Based on Biological ODEs (ICLR 2025 · similarity 0.88)
- Irregular Multivariate Time Series Forecasting: A Transformable Patching Graph Neural Networks Approach (ICML 2024 · similarity 0.87)
- S4M: S4 for multivariate time series forecasting with Missing values (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.