Research map / Video / Video-language understanding
Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Temporal Chain of Thought, an inference-only strategy where a single VLM iteratively selects relevant video frames conditioned on the question before answering, improving long-video QA.
Problem
Long-video question answering with VLMs, where long contexts full of irrelevant frames degrade accuracy and videos often exceed the model's context window.
Approach
Use the same instruction-tuned VLM as a frame selector: partition the video into segments, have the VLM output relevant frame indices and justifications per segment (Dynamic-Segment TCoT), then answer from the curated frames plus a few uniform frames, enabling inference-time scaling with a fixed context budget.
Key results
- On LVBench (videos ~68 min), TCoT with Gemini 1.5 Flash at 32K context reaches 61.7% vs 50.3% for baseline at 32K and 58.9% for baseline at 700K context.
- State-of-the-art on EgoSchema full set with 69.1% and on NExT-QA with 81.0%, with TCoT built on Gemini 1.5 Flash.
- TCoT improves GPT-4o-mini on LVBench from 48.0% to 53.5% and Qwen-2.5-VL-7B from 46.1% to 49.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard inference with Dynamic context aggregation | Gemini 1.5 Flash · LVBench | accuracy: 58.9 -> 61.7 | Helped |
| Replace standard inference with Dynamic context aggregation | Gemini 1.5 Flash · Egoschema subset | accuracy: 72.9 -> 75.2 | Helped |
| Replace standard inference with Dynamic context aggregation | Gemini 1.5 Flash · Egoschema full set | accuracy: 67.8 -> 69.1 | Helped |
| Replace standard inference with Dynamic context aggregation | Gemini 1.5 Flash · NExT-QA | accuracy: 80.0 -> 81.0 | Helped |
| Replace standard inference with Dynamic context aggregation | Gemini 1.5 Flash · OpenEQA | LLM Match: 68.0 -> 69.2 | Helped |
| Add high-confidence gating to independent segment answers (ablation) | Independent segment answer aggregation · LVBench | accuracy: 51.0 -> 55.6 | Helped |
| Use Dynamic context aggregation instead of independent segment answers (ablation) | Gemini 1.5 Flash · LVBench | accuracy: 55.6 -> 61.7 | Helped |
| Add zero-shot CoT prompting (ablation) | Gemini 1.5 Flash answer call · LVBench | accuracy: 49.5 -> 50.3 | Helped |
| Add two-stage prompting to zero-shot CoT (ablation) | Gemini 1.5 Flash answer call · LVBench | accuracy: 50.3 -> 49.4 | Hurt |
| Add self-consistency to zero-shot CoT (ablation) | Gemini 1.5 Flash answer call · LVBench | accuracy: 49.5 -> 51.7 | Helped |
| Vary segment size from s=64 to s=4, 16, 32, or 120 (ablation) | TCoT · Egoschema | accuracy: 75.2; 73.0; 73.6; 73.8; 73.8 | Mixed |
| Vary segment size from s=64 to s=4, 16, 32, or 120 (ablation) | TCoT · LVBench | accuracy: 57.8; 56.8; 57.0; 58.1; 57.8 | Mixed |
Surprising findings
- Self-consistency chain-of-thought gives only minimal gains on long-video QA, and pure text CoT variants yield marginal improvements on LVBench.
- Baseline Gemini inference saturates around 1000 frames on LVBench, so simply adding more frames does not help.
Limitations
- Requires a VLM with strong instruction-following to perform zero-shot frame selection.
- Low precision failures where too many irrelevant frames are selected, and low recall where key frames are missed.
- Gains are small or absent when the number of frames considered is low relative to video length.
Experimental setup
Datasets: EgoSchema LVBench OpenEQA NExT-QA
Benchmarks: EgoSchema LVBench NExT-QA OpenEQA
Metrics: accuracy LLM match score
Baselines: Gemini 1.5 Flash baseline inference Self-consistency CoT Feature similarity retrieval (SigLIP) VideoAgent VideoTree LLoVi LVNet Qwen-2.5-VL-7B baseline GPT-4o-mini baseline LangRepo MoreVQA BOLT
Builds on: Chain-of-Thought prompting self-consistency VideoAgent VideoTree
Closest papers by meaning
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.93)
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.92)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.91)
- SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding (ICLR 2025 · similarity 0.91)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.91)
- Not Another Text Benchmark: Putting the “Visual" Back in Visual Question Answering for Large Video Models (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.