Research map / Video / Video-language understanding
On the Consistency of Video Large Language Models in Temporal Comprehension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies prediction consistency of Video-LLMs in temporal grounding via shifted/rephrased grounding and holistic/compositional verification probes on new Charades-CON and ActivityNet-CON sets, and proposes event temporal verification tuning (VTune).
Problem
Video-LLMs can ground language queries to video moments, but their temporal comprehension is not robust: their moment predictions and answers to verification questions may be inconsistent with each other.
Approach
Probe consistency after an initial moment prediction using rephrased/shifted grounding and holistic/compositional verification probes, then tune models with event and temporal verification queries (VTune) that explicitly teach them to verify and correct query-moment alignment.
Key results
- Most open-source Video-LLMs score near chance (~50% relative) on holistic verification, while closed-source Gemini 1.5 Flash reaches 70.8% relative on Charades-CON holistic verification.
- VTune raises TimeChat's Charades-CON grounding R@1 IoU=0.5 from 55.8 to 76.2 with instruction tuning (w. IT) baseline comparison, and holistic verification from 16.7 to 44.8 shifted grounding.
- Prompting (CoT, description) gives unstable gains; instruction tuning boosts grounding but not consistency, e.g. TimeChat holistic verify improves only 2.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply VTune instead of instruction tuning | Video-LLaMA · Charades-CON | Ground: 45.1 -> 54.4 | Helped |
| Apply VTune instead of instruction tuning | TimeChat · Charades-CON | S-Ground: 10.5 -> 36.2 | Helped |
| Apply VTune instead of instruction tuning | TimeChat · ActivityNet-CON | Ground: 25.3 -> 37.4 | Helped |
| Add Chain-of-Thought prompting (ablation) | Video-LLaMA · Charades-CON | Ground: 14.2 -> 5.0 | Hurt |
| Add description prompting (ablation) | Video-LLaMA · Charades-CON | Ground: 14.2 -> 7.7 | Hurt |
| Add Chain-of-Thought prompting (ablation) | TimeChat · ActivityNet-CON | Ground: 4.6 -> 13.6 | Helped |
| Add description prompting (ablation) | TimeChat · ActivityNet-CON | Ground: 4.6 -> 5.9 | No clear effect |
| Add instruction tuning on target data (ablation) | Video-LLaMA · Charades-CON | S-Ground: 5.3 -> 8.9 | Hurt |
| Add event verification queries to grounding-query tuning (ablation) | Video-LLaMA · Charades-CON | Ground: 45.1 -> 48.8 | Helped |
| Add temporal verification queries to event-verification tuning (ablation) | Video-LLaMA · Charades-CON | S-Ground: 10.2 -> 10.9 | Helped |
| Add temporal verification queries to event-verification tuning (ablation) | TimeChat · Charades-CON | H-Verify: 40.4 -> 44.8 | Helped |
| Apply VTune | TimeChat · Charades-STA | R@1, IoU=0.5: +11.7 | Helped |
Surprising findings
- Time-aware models (VTimeLLM, VTG-LLM, TimeChat) show limited consistency gains over general-purpose models despite stronger grounding.
- Instruction tuning on target data improves grounding but can decrease consistency (e.g., Video-LLaMA shifted grounding).
Limitations
- Consistency evaluation restricted to correctly grounded queries (IoU>=0.5), and relies on GPT-4o-mini for query generation and answer evaluation.
- Evaluated only on two daily-activity datasets (Charades, ActivityNet), with average durations of 30 and 120 seconds.
Experimental setup
Datasets: Charades-STA ActivityNet-Captions
Benchmarks: Charades-CON ActivityNet-CON Charades-STA ActivityNet-Captions
Metrics: Recall@1 IoU=0.5 Recall@1 IoU=0.7 relative consistency score QA accuracy
Baselines: VideoChat2 Video-LLaVA Video-LLaMA Video-ChatGPT Video-LLaMA2 VTG-LLM VTimeLLM TimeChat GPT-4o Gemini 1.5 Flash BM-DETR Mr.BLIP
Builds on: instruction tuning chain-of-thought prompting TimeChat Charades-STA ActivityNet-Captions
Closest papers by meaning
- Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning (ICML 2024 · similarity 0.91)
- SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · similarity 0.90)
- Temporal Reasoning Transfer from Text to Video (ICLR 2025 · similarity 0.89)
- Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models (ICLR 2025 · similarity 0.89)
- Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization (NeurIPS 2025 · similarity 0.89)
- Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.