Research map / Video / Video-language understanding
Seq2Time: Sequential Knowledge Transfer for Video LLM Temporal Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Seq2Time trains time-sensitive video LLMs by turning image caption datasets and short Kinetics clip sequences into synthetic long sequences with relative position tokens, improving TimeChat on YouCook2 dense captioning and Charades-STA grounding.
Problem
Video LLMs lack temporal awareness because long videos with detailed captions and precise timestamp annotations are scarce and existing time-sensitive instruction datasets are small and low quality.
Approach
Use image sequences and concatenated short clip sequences, where sequence position serves as a free temporal label, with three image pretext tasks and a unified 4-digit relative position token shared between images and video timestamps, so temporal knowledge transfers from sequences to long videos.
Key results
- On YouCook2, TimeChat + Seq2Time reaches CIDEr 4.2 (vs 2.9 baseline, +44.8%) and F1 16.2 (vs 12.7, +27.6%).
- On Charades-STA, R@1 (IoU=0.7) rises from 11.7 to 13.7 (+17.1%) and R@1 (IoU=0.5) from 27.2 to 31.2.
- Even 5% of Seq2Time data yields notable gains, e.g. YouCook2 CIDEr 3.6 and Charades-STA R@1 (IoU=0.5) 26.8 vs 27.2 baseline for the grounding metric at 5%, with F1 rising to 15.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Seq2Time image- and clip-sequence training with unified relative position tokens | TimeChat · YouCook2 | F1: 12.7 -> 16.2 | Helped |
| Add Seq2Time image- and clip-sequence training with unified relative position tokens | TimeChat · Charades-STA | R@1 (IoU=0.7): 11.7 -> 13.7 | Helped |
| Add unified relative position tokens to Seq2Time (ablation) | TimeChat + Seq2Time · YouCook2 | F1: 15.7 -> 16.2 | Helped |
| Add clip sequence data (ablation) | TimeChat with image-sequence and more-video-caption data · YouCook2 | F1: 15.9 -> 16.2 | Helped |
| Add more video captions from ShareGPT4Video (ablation) | TimeChat with image-sequence data · YouCook2 | F1: 13.3 -> 15.9 | Helped |
| Add Indexed Image Captioning (ablation) | Seq2Time image-sequence training · YouCook2 | CIDEr: 2.6 -> 4.3 | Helped |
| Add Image Index Grounding (ablation) | Seq2Time image-sequence training · YouCook2 | F1: 11.9 -> 13.3 | Helped |
| Add Adjacent Location Reasoning (ablation) | Seq2Time image-sequence training · YouCook2 | CIDEr: 2.7 -> 4.3 | Helped |
| Add image sequence data (ablation) | Video-LLaMA · YouCook2 | F1: 0.2 -> 3.3 | Helped |
| Increase image- and clip-sequence data from 5% to 100% (ablation) | TimeChat + Seq2Time · YouCook2 | F1: 15.0 -> 16.2 | Helped |
| Add image sequence data (ablation) | TimeChat · YouCook2 | CIDEr: 2.9 -> 4.3 | Helped |
Surprising findings
- Adding image sequence data alone raises CIDEr by 48.3% over the baseline, suggesting temporal understanding can be learned from static image sequences.
- Adding more video captions (MC) substantially improves temporal localization but slightly degrades caption quality (CIDEr).
Limitations
- Baseline TimeChat reproduced with partial training data, yielding lower scores than originally reported.
- Object recognition errors and hallucinated events persist, attributed to visual encoder limitations.
- Clip captions depend on LongVA quality, which may limit caption accuracy.
Experimental setup
Datasets: LLaVA-ReCap COCO118K LLaVA-ReCap BLIP558K LLaVA-ReCap CC3M Kinetics-700 TimeIT Valley ShareGPT4Video YouCook2 Charades-STA
Benchmarks: YouCook2 Charades-STA
Metrics: SODA_c CIDEr METEOR F1 R@1 (IoU=0.5) R@1 (IoU=0.7) type-token ratio average caption length
Baselines: TimeChat VTimeLLM Momentor VTG-LLM
Builds on: TimeChat LLaVA-ReCap LongVA visual instruction tuning
Closest papers by meaning
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (CVPR 2024 · similarity 0.89)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.89)
- SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · similarity 0.89)
- Temporal Reasoning Transfer from Text to Video (ICLR 2025 · similarity 0.89)
- TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning (ICLR 2025 · similarity 0.89)
- SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.