Research map / Video / Video-language understanding
Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MotionEpic, a video MLLM that encodes and generates video spatial-temporal scene graphs (STSGs), and a five-step Video-of-Thought (VoT) chain-of-thought framework that reasons from pixel-level grounding to commonsense cognition for complex video QA.
Problem
Complex video understanding and reasoning requires both fine-grained spatial-temporal perception and cognitive-level commonsense reasoning, which existing video MLLMs and CoT prompting do not adequately provide.
Approach
Integrates STSG representations into a video MLLM for pixel-level grounding (via grounding-aware tuning) and applies a multi-step CoT-style decomposition (target identification, tracking, action analysis, ranking-based QA, verification) to video QA.
Key results
- On NExT-QA, VoT with MotionEpic reaches 76.0 Acc@All versus 75.5 for the prior best VLAP.
- On Causal-VidQA hard subset, MotionEpic with VoT scores 74.3 on Acc@P-A versus 80.6 human performance, far above CoT Video-LLaVA's 32.4 on the same subset.
- Zero-shot average across four QA datasets rises from 57.8 (CoT) to 62.3 (VoT) for MotionEpic.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply VoT reasoning instead of CoT to MotionEpic | MotionEpic · Causal-VidQA Description | accuracy: 78.5 -> 81.2 | Helped |
| Apply VoT reasoning instead of CoT to MotionEpic | MotionEpic · NExT-QA All | accuracy: 72.2 -> 76.0 | Helped |
| Apply VoT reasoning instead of CoT to MotionEpic | MotionEpic · Zero-shot AVG. | accuracy: 57.8 -> 62.3 | Helped |
| Remove grounding verification (Verify-G) (ablation) | MotionEpic + VoT · Zero-shot AVG. | accuracy: 62.3 -> 59.0 | Hurt |
| Remove commonsense verification (Verify-C) (ablation) | MotionEpic + VoT · Zero-shot AVG. | accuracy: 62.3 -> 60.1 | Hurt |
| Remove grounding-aware tuning objective L2 (generate the whole STSG from video) (ablation) | MotionEpic · Zero-shot video QA | accuracy | Hurt |
| Remove grounding-aware tuning objective L4 (describe actions and output the corresponding tracklet) (ablation) | MotionEpic · Zero-shot video QA | accuracy | Hurt |
Surprising findings
- CoT prompting gives only limited improvement for Video-LLaVA on video reasoning tasks.
- Implicit integration of scene graph features in the MLLM outperforms explicit STSG text integration (Video-LLaVA+STSG).
Limitations
- Relies on STSG annotations and an external STSG parser for grounding-aware tuning data, and the VoT pipeline requires multiple sequential LLM calls.
- Evaluation focuses on video QA; generalization to other video tasks is claimed but not tested.
- Broader impact statement notes potential for malicious misuse and high energy consumption.
Experimental setup
Datasets: Action Genome WebVid-10M VideoChat instruction data Video-ChatGPT instruction data VLEP STAR IntentQA Social-IQ Causal-VidQA NExT-QA MSR-VTT ActivityNet
Benchmarks: VLEP STAR IntentQA Social-IQ Causal-VidQA NExT-QA MSR-VTT ActivityNet Action Genome
Metrics: accuracy
Baselines: InternVideo LLaMA-VQA VLAP SeViLA TranSTR HiTeA VideoChat2 Video-LLaMA Video-ChatGPT VideoChat Video-LLaVA GPS-Net
Builds on: Chain-of-Thought prompting Video-LLaVA Video-ChatGPT VideoChat Video-LLaMA BLIP-2 Q-Former LoRA
Closest papers by meaning
- When Thinking Drifts: Evidential Grounding for Robust Video Reasoning (NeurIPS 2025 · similarity 0.91)
- VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning? (ICLR 2026 · similarity 0.90)
- Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level (CVPR 2025 · similarity 0.90)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.90)
- Look, Remember and Reason: Grounded Reasoning in Videos with Language Models (ICLR 2024 · similarity 0.90)
- TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.