Research map / Video / Video-language understanding
VTimeLLM: Empower LLM to Grasp Video Moments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VTimeLLM, a Video LLM with a three-stage boundary-aware training (image-text alignment, multi-event video QA with LoRA, and instruction tuning) to ground and caption fine-grained video moments.
Problem
Existing Video LLMs give coarse whole-video descriptions and fail to localize precise start and end times of events, limiting temporal grounding, dense captioning and temporal reasoning.
Approach
A boundary-aware three-stage training strategy that inserts a boundary perception stage (multi-event video QA with timestamp tokens 00-99 on InternVid-10M-FLT subset, trained via LoRA) between image-text feature alignment and LLM-generated high-quality instruction tuning, so the LLM learns to output frame-index time boundaries.
Key results
- VTimeLLM-7B reaches 44.0 R@0.3 and 30.4 mIoU on ActivityNet temporal grounding, versus 26.4 R@0.3 and 18.9 mIoU for VideoChatGPT-7B.
- On ActivityNet dense captioning, VTimeLLM-7B obtains 5.8 SODA_c and 27.6 CIDEr, compared with 1.9 and 5.8 for VideoChatGPT-7B.
- On the Video-ChatGPT dialogue benchmark, VTimeLLM achieves a mean score of 2.85, above BT-Adapter (2.69) and VideoChatGPT (2.38).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VTimeLLM's three-stage training rather than VideoChatGPT-7B | VTimeLLM · ActivityNet temporal grounding | mIoU: 18.9 -> 30.4 | Helped |
| Use VTimeLLM's three-stage training rather than VideoChatGPT-7B | VTimeLLM · Charades-STA temporal grounding | mIoU: 13.7 -> 31.2 | Helped |
| Use VTimeLLM's three-stage training rather than VideoChatGPT-7B | VTimeLLM · ActivityNet dense video captioning | CIDEr: 5.8 -> 27.6 | Helped |
| Scale VTimeLLM from 7B to 13B parameters | VTimeLLM · ActivityNet temporal grounding | R@0.5: 27.8 -> 29.5 | Helped |
| Scale VTimeLLM from 7B to 13B parameters | VTimeLLM · Charades-STA temporal grounding | R@0.5: 27.5 -> 34.3 | Helped |
| Use image rather than video data for Stage 1 feature alignment (ablation) | VTimeLLM · ActivityNet temporal grounding | R@0.3: 33.5 -> 36.1 | Helped |
| Use image data rather than combined image and video data for Stage 1 alignment (ablation) | VTimeLLM · ActivityNet temporal grounding | R@0.3: 31.3 -> 36.1 | Helped |
| Tune rather than freeze the visual adapter after Stage 1 (ablation) | VTimeLLM · ActivityNet temporal grounding | R@0.3: 36.1 -> 35.6 | No clear effect |
| Add a new LoRA module in Stage 3 rather than reuse the Stage 2 LoRA (ablation) | VTimeLLM · ActivityNet temporal grounding | R@0.3: 39.3 -> 44.0 | Helped |
| Remove Stage 1 feature alignment (ablation) | VTimeLLM · ActivityNet dense video captioning | CIDEr: 17.0 -> 10.2 | Hurt |
| Remove Stage 2 boundary perception training (ablation) | VTimeLLM · ActivityNet temporal grounding | R@0.3: 39.3 -> 31.0 | Hurt |
| Add Stage 3 instruction tuning (ablation) | VTimeLLM · ActivityNet temporal grounding | R@0.3: 36.1 -> 44.0 | Helped |
Surprising findings
- Training without the stage 1 image alignment yields inflated ActivityNet grounding scores because the model predicts near-whole-video segments, with IoU roughly equal to the ground-truth duration ratio.
- Template-based QA in the second stage caused overfitting, so the authors switched to LLM-generated dialogues for stage 3.
Limitations
- Stage 2 data relies on automatically generated, noisy InternVid annotations.
- Frame-index time tokens are limited to 100 frames sampled from each video, which restricts temporal resolution for long videos.
- Stage 3 uses a relatively small set of about 36k dialogues, and the model struggles on Charades-STA without training on it.
Experimental setup
Datasets: LCS-558K InternVid-10M-FLT ActivityNet Captions DiDeMo VideoInstruct100K Charades-STA
Benchmarks: ActivityNet Captions Charades-STA Video-ChatGPT generative performance benchmark
Metrics: R@0.3 R@0.5 R@0.7 mIoU SODA_c CIDEr METEOR correctness/detail/contextual/temporal/consistency scores
Baselines: VideoChat VideoLLaMA VideoChatGPT LLaMA-Adapter BT-Adapter
Builds on: LLaVA Video-ChatGPT Vicuna LoRA
Closest papers by meaning
- Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning (ICML 2024 · similarity 0.92)
- SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · similarity 0.91)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.90)
- Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.90)
- Temporal Reasoning Transfer from Text to Video (ICLR 2025 · similarity 0.90)
- SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.