Research map / Video / Video-language understanding
VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds VideoEspresso, an automatically generated video QA dataset with multimodal chain-of-thought (CoT) evidence and bounding boxes, plus a hybrid tiny-selector plus LVLM framework that selects core frames and reasons with two-stage SFT.
Problem
Video reasoning with LVLMs is limited by the scarcity of large, high-quality VideoQA datasets; manual annotation is costly and coarse, while frame-by-frame automatic construction is redundant and computationally expensive.
Approach
Semantic-aware redundancy removal over frame captions, GPT-4o-based QA and multimodal CoT annotation (GroundingDINO boxes plus caption retrieval for temporal grounding), and a tiny LVLM+LLM selector that picks question-relevant core frames before a two-stage evidence-then-answer SFT reasoning LVLM.
Key results
- The proposed model reaches 34.1% average accuracy on the objective benchmark, versus 28.7% for InternVL2-8B and 26.4% for GPT-4o.
- Ablation: adding ground-truth CoT raises accuracy from 34.13 to 72.95; removing CoT drops it to 31.32.
- Selector with 1B/0.5B reduces input frames to 2.36 on average (vs. 16 uniform for many baselines) at TFLOPs of 9.26 vs. 62.78 for LLaVA-N-inter, while keeping accuracy comparable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ground-truth CoT (ablation) | VideoEspresso reasoning model · VideoEspresso benchmark | Accuracy: 34.13 -> 72.95 | Helped |
| remove bounding boxes from CoT evidence (ablation) | VideoEspresso reasoning model · VideoEspresso benchmark | Accuracy: 34.13 -> 33.14 | Hurt |
| remove the CoT process (ablation) | VideoEspresso reasoning model · VideoEspresso benchmark | Accuracy: 34.13 -> 31.32 | Hurt |
| replace uniform frame sampling with ground-truth core-frame selection (ablation) | VideoEspresso reasoning model · VideoEspresso benchmark | Accuracy: 33.74 -> 37.54 | Helped |
| use InternVL2-1B and QwenLM-1.5B for frame selection instead of uniform sampling (ablation) | VideoEspresso reasoning model · VideoEspresso benchmark | Accuracy: 33.74 -> 34.76 | Helped |
| use InternVL2-1B and QwenLM-0.5B for frame selection instead of uniform sampling (ablation) | VideoEspresso reasoning model · VideoEspresso benchmark | Accuracy: 33.74 -> 34.13 | Helped |
Surprising findings
- Lightweight LVLMs such as LLaVA-Next-interleave-0.5B and Qwen-VL-1.5B underperformed as selectors, attributed to processing too many image tokens.
- Plugging the core-frame selector into GPT-4o, InternVL2, and LongVA is mostly zero-shot and gives mixed results, with LongVA and LLaVA-N-i slightly losing accuracy despite large frame reductions.
Limitations
- Evaluation relies on GPT-4o-based open-ended judging and a similarity-with-distractors heuristic, which may be noisy.
- Auto-generated data depends on GPT-4o and frame captioning models, so errors and hallucinations can propagate into QA and CoT annotations.
Experimental setup
Datasets: XD-Violence Vatex CUVA SQA3D MovieChat Storystream Youcook2 MVBench
Benchmarks: VideoEspresso benchmark (14 tasks, objective) VideoEspresso subjective benchmark
Metrics: accuracy subjective score on logic, factuality, description accuracy,
Baselines: GPT-4o Qwen-VL-Max LLaVA-1.5 InternVL2 LLaVA-N-Inter Qwen2-VL LongVA-DPO mPLUG-Owl3 LLaVA-N-Video
Builds on: LLaVA-NeXT-interleave GroundingDINO BGE-M3 InternVL2
Closest papers by meaning
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.91)
- Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task (NeurIPS 2025 · similarity 0.91)
- Divide and Conquer: Exploring Language-centric Tree Reasoning for Video Question-Answering (ICML 2025 · similarity 0.91)
- Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning (CVPR 2025 · similarity 0.91)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.91)
- Not Another Text Benchmark: Putting the “Visual" Back in Visual Question Answering for Large Video Models (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.