Research map / Video / Video-language understanding
TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts the short-video MLLM VideoChat2 into VideoChat-T for long videos via token shuffle compression, a temporal adaptive position encoding (TAPE), and grounding-centric instruction tuning on the 349K-instance TimePro dataset including a new Temporal Grounded Caption task.
Problem
Short-form video MLLMs struggle with long video understanding, and prior temporal-grounding MLLMs lose general video QA ability while long-context or token-compression methods lack fine-grained temporal localization.
Approach
Compress long video tokens by channel-wise token shuffling initialized to mean pooling, add a zero-padded convolutional temporal position adapter (TAPE), and tune the model on grounding-centric instructions (incl. Temporal Grounded Caption) so that explicit timestamp prediction anchors answers to relevant segments.
Key results
- VideoChat-T reaches 48.7 R@1 (IoU=0.5) zero-shot on Charades-STA, versus 32.2 for TimeChat, and 67.1 after fine-tuning on the benchmark's training set.
- VideoChat-T improves over VideoChat2 by 5.6 points on Egoschema full (60.0) and 6.8 points on VideoMME w/o subs (46.3).
- VideoChat-T uses 3 tokens per frame, with about 5.1% of LLaVA-OneVision's FLOPs at 128 frames and 0.63 s inference on a single A100, but drops 0.5 points on MVBench (59.9 vs 60.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| upgrade VideoChat2 with TimeSuite | VideoChat2 · Egoschema full | accuracy: 54.4 -> 60.0 (+5.6) | Helped |
| upgrade VideoChat2 with TimeSuite | VideoChat2 · VideoMME w/o subtitles | accuracy: 39.5 -> 46.3 (+6.8) | Helped |
| upgrade VideoChat2 with TimeSuite | VideoChat2 · MVBench | accuracy: 60.4 -> 59.9 (-0.5) | Hurt |
| upgrade VideoChat2 with TimeSuite | VideoChat2 · Charades-STA R@1 IoU=0.5 | R@1: +16.5 points | Helped |
| upgrade VideoChat2 with TimeSuite | VideoChat2 · QVHighlights mAP | mAP: +13.0 points | Helped |
| add TAPE (ablation) | VideoChat-T · Egoschema full | accuracy: 59.1 -> 60.0 | Helped |
| add TAPE (ablation) | VideoChat-T · VideoMME w/o subtitles | accuracy: 45.9 -> 46.3 | Helped |
| add TAPE (ablation) | VideoChat-T · Charades-STA R@1 IoU=0.5 | R@1: 47.1 -> 48.7 | Helped |
| freeze TAPE during the first epoch (ablation) | VideoChat-T · Egoschema full | accuracy: 59.0 -> 60.0 | Helped |
| unfreeze TAPE in the first epoch (ablation) | VideoChat-T · Charades-STA R@1 IoU=0.5 | R@1: 48.7 -> 52.4 | Helped |
| replace Token Shuffle with pooling (ablation) | VideoChat-T · Charades-STA R@1 IoU=0.5 | R@1: 48.7 -> 40.3 | Hurt |
| replace Token Shuffle with clustering (ablation) | VideoChat-T · QVHighlights HIT@1 | HIT@1: 54.1 -> 40.1 | Hurt |
Surprising findings
- Fine-tuning VideoChat-T on the QVHighlights training set yields almost no mAP gain (26.5 to 27.0), which the authors attribute to LLM numerical output limits.
- VideoChat2 degrades sharply when its input frame count is raised from 16 to 128 (MVBench 60.4 to 42.1), indicating architecture-data mismatch rather than only information loss.
Limitations
- Highlight detection shows little gain from fine-tuning because LLMs struggle to output many numeric saliency scores.
- Short-video MVBench performance drops slightly (0.5 points) relative to VideoChat2 after incremental fine-tuning.
Experimental setup
Datasets: TimePro (DiDeMo, QuerYD, HiREST, ActivityNet-Captions, ViTT, YouCook2, TVSum, SumMe, COIN, YT-Temporal, ActivityNet-RTL, VideoChatGPT VideoChat EgoQA MovieChat-QA STAR MovieChat-Caption
Benchmarks: Charades-STA QVHighlights Egoschema VideoMME MVBench
Metrics: R@1 IoU=0.3 R@1 IoU=0.5 R@1 IoU=0.7 mAP HIT@1 accuracy
Baselines: TimeChat VideoChat2 MovieChat HawkEye VTimeLLM ChatVTG GroundingGPT LLaVA-OneVision Qwen2-VL LLAMA-Vid Chat-UniVi MovieChat+
Builds on: VideoChat2 CPVT TimeChat UMT
Closest papers by meaning
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (CVPR 2024 · similarity 0.93)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.91)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.90)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.90)
- Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding (NeurIPS 2025 · similarity 0.90)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.