Research map / Video / Video-language understanding
TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TimeChat, a time-sensitive video LLM with a timestamp-aware frame encoder and sliding video Q-Former, trained by instruction tuning on the 125K-instance TimeIT dataset.
Problem
Existing video LLMs compress long videos into a fixed number of tokens and fail to associate visual content with accurate timestamps, hurting temporal localization in long videos.
Approach
Bind each frame's visual tokens to its timestamp text inside the image Q-Former, and use a sliding-window video Q-Former so the number of video tokens scales with video length (constant compression rate), followed by instruction tuning on a time-aware dataset (TimeIT).
Key results
- Zero-shot TimeChat (7B) reaches 12.6 F1 and 3.4 CIDEr on YouCook2, improving on the best 7B baseline by +9.2 F1 and +2.8 CIDEr.
- Zero-shot R@1 (IoU=0.5) on Charades-STA is 32.2, +27.5 over Valley; HIT@1 on QVHighlights is 23.9, +5.8 over the best prior VidLLM.
- Ablation on YouCook2: removing the sliding video Q-Former drops F1 by 3.0 and CIDEr by 2.8; removing the timestamp-aware frame encoder drops F1 by 2.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add sliding video Q-Former (ablation) | TimeChat · YouCook2 | SODA_c: +1.0 | Helped |
| add sliding video Q-Former (ablation) | TimeChat · YouCook2 | CIDEr: +2.8 | Helped |
| add sliding video Q-Former (ablation) | TimeChat · YouCook2 | F1 Score: +3.0 | Helped |
| add timestamp-aware frame encoder (ablation) | TimeChat · YouCook2 | SODA_c: +0.3 | Helped |
| add timestamp-aware frame encoder (ablation) | TimeChat · YouCook2 | CIDEr: +1.4 | Helped |
| add timestamp-aware frame encoder (ablation) | TimeChat · YouCook2 | F1 Score: +2.3 | Helped |
Surprising findings
- Multi-model pipelines (InstructBLIP+ChatGPT, VideoChat-Text) outperform end-to-end VidLLMs on highlight detection on QVHighlights.
- After fine-tuning on specialized data, TimeChat still lags specialized task-specific models such as QD-DETR, MMN and VDI, though it is a generalist.
Limitations
- Still trails specialized models on temporal tasks after fine-tuning.
- Outputs require heuristic rules to parse LLM responses, which may be brittle.
- Richness and detail of generated summaries remain limited, per the authors.
Experimental setup
Datasets: TimeIT Valley ActivityNet Captions ViTT YouCook2 DiDeMo QuerYD Charades-STA HiREST TVSum SumMe QVHighlights
Benchmarks: YouCook2 QVHighlights Charades-STA
Metrics: SODA_c CIDEr F1 Score mAP HIT@1 R@1 (IoU=0.5) R@1 (IoU=0.7)
Baselines: VideoChat-Embed Video-LLaMA Valley VideoChat-Text InstructBLIP+ChatGPT Vid2Seq QD-DETR MMN VDI
Builds on: InstructBLIP Video-LLaMA VideoChat Vid2Seq
Closest papers by meaning
- TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning (ICLR 2025 · similarity 0.93)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.91)
- MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (CVPR 2024 · similarity 0.91)
- Temporal Reasoning Transfer from Text to Video (ICLR 2025 · similarity 0.90)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.90)
- MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.