Research map / Video / Video-language understanding
Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Deep Video Discovery (DVD), an LLM agent that performs adaptive agentic search over a multi-granular video database using Global Browse, Clip Search, and Frame Inspect tools, reaching 74.2% on LVBench (76.0% with transcripts).
Problem
Long-form (hour-long) video question answering is hard because of extended temporal-spatial context and information density; prior video agents follow hand-designed fixed workflows that don't adapt to different queries.
Approach
Instead of prescribing a fixed search workflow, the agent is given search-centric tools at three granularities (global, clip caption, frame level) over a pre-built multi-granular database and lets a reasoning LLM (o3) autonomously decide tool sequences in an observe-reason-act loop.
Key results
- Achieves 74.2% accuracy on LVBench, surpassing the prior best MR. Video (60.8%) and base OpenAI o3 (57.1%); 76.0% with auxiliary transcripts.
- Reaches 67.3% on the Video-MME Long subset (w/o subtitles) and 76.6% on EgoSchema, the latter above the previous best VCA (73.6%).
- Ablation: removing Clip Search causes the largest drop (12.3%), and replacing the adaptive workflow with VideoAgent's hand-crafted workflow lowers accuracy by 7.5 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add auxiliary transcripts (ablation) | Deep Video Discovery · LVBench | accuracy: 74.2 -> 76.0 | Helped |
| replace GPT-4.1 with GPT-4.1-mini for database construction (ablation) | Deep Video Discovery · LVBench with transcripts | accuracy: 76.0 -> 71.9 | Hurt |
| replace OpenAI o3 with GPT-4.1-mini for Frame Inspect (ablation) | Deep Video Discovery · LVBench with transcripts | accuracy: 76.0 -> 72.3 | Hurt |
| replace OpenAI o3 with OpenAI o4-mini for reasoning (ablation) | Deep Video Discovery · LVBench with transcripts | accuracy: 76.0 -> 70.2 | Hurt |
| replace OpenAI o3 with GPT-4o for reasoning (ablation) | Deep Video Discovery · LVBench with transcripts | accuracy: 76.0 -> 62.3 | Hurt |
| replace GPT-4.1 with OpenAI o3 for database construction (ablation) | Deep Video Discovery · LVBench with transcripts | accuracy: 76.0 -> 71.9 | Hurt |
| remove Global Browse (ablation) | Deep Video Discovery · LVBench with transcripts | accuracy: 71.9 -> 69.0 | Hurt |
| remove Frame Inspect (ablation) | Deep Video Discovery · LVBench with transcripts | accuracy: 71.9 -> 63.5 | Hurt |
| remove Clip Search (ablation) | Deep Video Discovery · LVBench with transcripts | accuracy: 71.9 -> 59.6 | Hurt |
| use DeepSeek-R1 as the reasoning model (ablation) | Deep Video Discovery · LVBench | accuracy: 76.0 -> 68.5 | Hurt |
| use DeepSeek-V3 as the reasoning model (ablation) | Deep Video Discovery · LVBench | accuracy: 76.0 -> 57.5 | Hurt |
| use Qwen3-32B-Thinking as the reasoning model (ablation) | Deep Video Discovery · LVBench | accuracy: 76.0 -> 57.3 | Hurt |
Surprising findings
- GPT-4o collapses into a short 'Simple Action' pattern on 91.4% of queries, averaging only 4.6 steps, which drives its low accuracy in the agent setting.
- Swapping in open-source reasoning models (DeepSeek-R1 68.5%, Qwen3-32B 57.3%) still yields results competitive with or above GPT-4o/o3 baselines.
Limitations
- Iterative reasoning incurs higher computational overhead.
- Azure OpenAI content filtering blocks a portion of benchmark queries, affecting both the baseline and DVD results.
- Agent performance depends strongly on the reasoning model; weaker models collapse into premature answering or search loops.
Experimental setup
Datasets: LVBench LongVideoBench Video-MME EgoSchema
Benchmarks: LVBench LongVideoBench Video-MME EgoSchema
Metrics: accuracy
Baselines: OpenAI o3 GPT-4o Gemini-1.5-Pro Gemini-2.0-Flash MR. Video VCA VideoTree VideoAgent AdaReTaKe VideoChat-Flash Qwen2.5-VL-72B InternVL2.5-78B
Builds on: ReAct VideoAgent VideoTree
Closest papers by meaning
- VideoSeek: Long-Horizon Video Agent with Tool-Guided Seeking (CVPR 2026 · similarity 0.92)
- Think, Then Verify: A Hypothesis-Verification Multi-Agent Framework for Long Video Understanding (CVPR 2026 · similarity 0.90)
- Hierarchical Long Video Understanding with Audiovisual Entity Cohesion and Agentic Search (CVPR 2026 · similarity 0.89)
- Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA (NeurIPS 2026 · similarity 0.88)
- VideoLucy: Deep Memory Backtracking for Long Video Understanding (NeurIPS 2025 · similarity 0.87)
- ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.