Research map / Time series / Foundation models for time series
ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Time-Series QA task with the EngineMT-QA dataset (aero engine signals) and ITFormer, a lightweight connector that fuses a time-series encoder with frozen LLMs via learnable instruct tokens and instruct time attention.
Problem
Integrating high-dimensional multivariate time-series with natural language for interactive question answering has no standardized benchmark and few methods that model temporal-textual interaction beyond auxiliary features.
Approach
A query-conditioned fusion module (learnable instruct tokens, channel-then-time instruct attention, and position encodings over steps, channels, and segments) that compresses a PatchTST encoder's outputs into LLM-compatible tokens, training only the ~0.07% alignment parameters while the LLM and encoder stay frozen.
Key results
- ITFormer-7B reaches 58.04 Rouge-L and 38.23 BLEU on Understanding and 88.69 Accuracy on Reasoning, the best across the four EngineMT-QA tasks versus baselines such as MCAN-VQA (48.52 Rouge-L, 54.78 Accuracy).
- On TimeSeriesExam, ITFormer reaches 0.79 accuracy on Causality Analysis, above GPT-4o (0.68) and MCAN-VQA (0.73); after transfer from EngineMT-QA it reaches 0.86 on Pattern and 0.89 on Anomaly.
- Ablation: full ITFormer (all four components) reaches 68.21% accuracy and 22.74 BLEU, versus 49.82% and 16.41 for the base configuration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ITA only (ablation) | ITFormer without auxiliary modules · EngineMT-QA component ablation | Average accuracy: 49.82 -> 50.94 | Helped |
| add TPE only (ablation) | ITFormer without auxiliary modules · EngineMT-QA component ablation | Average accuracy: 49.82 -> 54.16 | Helped |
| add TAL only (ablation) | ITFormer without auxiliary modules · EngineMT-QA component ablation | Average BLEU: 16.41 -> 13.86 | Hurt |
| add LIT only (ablation) | ITFormer without auxiliary modules · EngineMT-QA component ablation | Average accuracy: 49.82 -> 53.69 | Helped |
| add ITA and TPE (ablation) | ITFormer without auxiliary modules · EngineMT-QA component ablation | Average accuracy: 49.82 -> 60.29 | Helped |
| add TAL to ITA and TPE (ablation) | ITFormer with ITA and TPE · EngineMT-QA component ablation | Average accuracy: 60.29 -> 62.17 | Helped |
| add LIT to ITA, TPE, and TAL (ablation) | ITFormer with ITA, TPE, and TAL · EngineMT-QA component ablation | Average accuracy: 62.17 -> 68.21 | Helped |
| increase ITFormer layers beyond 2 (ablation) | ITFormer · EngineMT-QA | performance | No clear effect |
| vary LIT length away from 25 tokens (ablation) | ITFormer · EngineMT-QA | performance | Hurt |
| pretrain on EngineMT-QA before fine-tuning | ITFormer · TimeSeriesExam | Accuracy: 0.83 -> 0.86 (Pattern Recognition); 0.84 -> 0.89 (Anomaly Detection); 0.95 -> 0.98 (Noise Understanding); 0.92 -> 0.94 ( | Helped |
Surprising findings
- Multimodal APIs GPT-4o and Gemini score poorly on EngineMT-QA, with ChatGPT-4o Decision BLEU at 3.30, and they were fed plotted signal images rather than raw series.
Limitations
- Dataset size is inconsistent (abstract says over 110k QA pairs, body says 11k) and the dataset is built from one domain (aero engines) using LLM-generated and expert-refined QA.
- Baselines were adapted by replacing their image encoders with the PatchTST encoder and some were run on plotted signal images, so comparisons may be confounded.
- Ablation and efficiency results are reported on a single run with no variance estimates.
Experimental setup
Datasets: EngineMT-QA N-CMAPSS TimeSeriesExam
Benchmarks: EngineMT-QA TimeSeriesExam
Metrics: Accuracy F1 Rouge-L BLEU
Baselines: GPT-4o Gemini InstructBLIP MCAN-VQA CoCa Time-LLM AutoTime
Builds on: PatchTST Time-LLM InstructBLIP Qwen2.5
Closest papers by meaning
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.87)
- TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loop (NeurIPS 2025 · similarity 0.87)
- ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data (AAAI 2025 · similarity 0.86)
- TimeCAP: Learning to Contextualize, Augment, and Predict Time Series Events with Large Language Model Agents (AAAI 2025 · similarity 0.86)
- TimeCMA: Towards LLM-Empowered Multivariate Time Series Forecasting via Cross-Modality Alignment (AAAI 2025 · similarity 0.86)
- Position: What Can Large Language Models Tell Us about Time Series Analysis (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.