Research map / Video / Video-language understanding
Temporal Reasoning Transfer from Text to Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that Video LLM temporal reasoning bottlenecks lie in the LLM decoder, and proposes T3, which synthesizes text-only temporal QA from image-caption data to fine-tune LongVA-7B and improve video temporal understanding without any video data.
Problem
Video LLMs struggle with temporal reasoning (order, attribute change, referring, grounding). Prior work blamed visual encoding, but it is unclear whether the vision encoder or the LLM decoder is the bottleneck, and video instruction data for temporal reasoning is scarce.
Approach
Probing diagnostics show visual features already encode temporal information while the LLM decoder fails on textual temporal QA; so temporal reasoning is transferred by fine-tuning the LLM on text-only temporal QA synthesized from image-caption sequences with distractor captions.
Key results
- T3 raises LongVA-7B TempCompass average from 55.9 to 61.2 without any video training, above ShareGPT4Video-8B (59.4) trained on 28k videos.
- T3 w/ LLaVA-Next reaches 49.7 on Video-MME temporal reasoning (from 37.3), surpassing VILA1.5-40B (40.7) and InternVL-Chat-V1.5-20B (33.3).
- Textual validation accuracy correlates with TempCompass (Pearson r=0.89) and MLVU (r=0.85).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train LongVA-7B on text-only T3 temporal tasks | LongVA-7B · TempCompass | Average accuracy: 55.9 -> 61.2 | Helped |
| Continue fine-tuning on LLaVA-Next data | LongVA-7B · TempCompass | Average accuracy: 55.9 -> 56.0 | No clear effect |
| Fine-tune on HotpotQA | LongVA-7B · TempCompass | Average accuracy: 55.9 -> 54.9 | Hurt |
| Train on Temporal Change (1x), combining Order and Attribute (ablation) | LongVA-7B · TempCompass | Order accuracy: 54.3 -> 65.2 | Helped |
| Train on Temporal Change (1x), combining Order and Attribute (ablation) | LongVA-7B · TempCompass | Attribute Change accuracy: 51.7 -> 63.9 | Helped |
| Increase distractor captions from 1x to 8x (ablation) | Temporal Change training for LongVA-7B · TempCompass | Average accuracy: 59.5 -> 61.0 | Helped |
| Add Temporal Grounding training (ablation) | LongVA-7B · TempCompass | Average accuracy: 55.9 -> 41.0 | Hurt |
| Combine Temporal Change (1x) with Order-Template (ablation) | LongVA-7B · TempCompass | Average accuracy: 59.5 -> 60.6 | Helped |
| Add distractor captions to Temporal Change training (ablation) | LongVA-7B · TempCompass | Average accuracy: 59.0 -> 59.5 | Helped |
| Use captions instead of images for Order transfer (ablation) | LongVA-7B · TempCompass | Order accuracy: 55.4 -> 63.6 | Helped |
| Mix T3 with LLaVA-Next training data | LongVA-7B · MLVU | Macro Average accuracy: 56.4 -> 58.1 | Helped |
| Mix T3 with LLaVA-Next training data | LongVA-7B · Video-MME | Temporal Reasoning accuracy: 37.3 -> 49.7 | Helped |
Surprising findings
- Visual probes on Video LLM embeddings reach near-perfect accuracy while the LLM decoder, given textual frame captions, performs poorly, implicating the LLM rather than the vision encoder.
- Training on HotpotQA or the original LLaVA-Next instruction data degrades or barely changes temporal performance.
Limitations
- Covers only four temporal dimensions, an approximation of temporal reasoning.
- Synthesized data may be too simple for stronger LLMs, limiting gains.
Experimental setup
Datasets: LLaVA-ReCap-558K LLaVA-Next instruction data HotpotQA
Benchmarks: TempCompass MLVU Video-MME synthesized temporal probing benchmark
Metrics: accuracy Pearson correlation
Baselines: LLaVA-Next continued fine-tuning HotpotQA fine-tuning ShareGPT4Video-8B LLaVA-NeXT-Video-7B-DPO VILA1.5-40B InternVL-Chat-V1.5-20B GPT-4o Video-ChatGPT Video-LLaVA MA-LMM
Builds on: LongVA LLaVA-NeXT ShareGPT4Video TempCompass
Closest papers by meaning
- SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · similarity 0.91)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.91)
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (CVPR 2024 · similarity 0.90)
- Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning (ICML 2024 · similarity 0.90)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.90)
- Not Another Text Benchmark: Putting the “Visual" Back in Visual Question Answering for Large Video Models (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.