Research map / Video / Video-language understanding
DrVideo: Document Retrieval Based Long Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Converts long-video understanding into long-document retrieval and understanding, using a captioner-built text document, retrieval of key frames, and a planning/interaction agent loop with CoT answering, training-free.
Problem
Long video understanding is hard because key frames are difficult to locate among many frames and long-range reasoning over many events is required, while video-LMs cannot ingest whole long videos.
Approach
Recasts long video understanding as long-document retrieval: each frame is captioned into a text document, top-K question-relevant frames are retrieved by embedding similarity, and an agent loop iteratively requests augmented detailed descriptions of missing frames before a zero-shot chain-of-thought answer.
Key results
- On EgoSchema subset, DrVideo with GPT-4 reaches 66.4% accuracy versus 62.8% for VideoAgent and 61.2% for LLoVi (GPT-4).
- On MovieChat-1K global mode, DrVideo achieves 93.1% accuracy, and 56.4% in breakpoint mode.
- On Video-MME long split with subtitles, DrVideo reaches 71.7% accuracy; without subtitles it reaches 51.7%, above Claude 3.5 Sonnet (51.2%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the retrieval module (ablation) | DrVideo · EgoSchema subset | accuracy: 57.4% to 60.6% | Helped |
| Add the multi-stage agent interaction loop (ablation) | DrVideo · EgoSchema subset | accuracy: 60.6% to 62.6% | Helped |
| Add chain-of-thought answering (ablation) | DrVideo · EgoSchema subset | accuracy: 62.2% to 62.6% | Helped |
| Use both VQA and caption augmentation rather than either type alone (ablation) | DrVideo · EgoSchema subset | accuracy: 60.4% to 62.6%; 61.8% to 62.6% | Helped |
| Increase initial retrieved frames from Top-K 5 to Top-K 10 or 20 (ablation) | DrVideo · EgoSchema subset | accuracy: 62.6% to 61.4%; 62.6% to 60.6% | Hurt |
| Use more than two iterative interaction rounds (ablation) | DrVideo · EgoSchema | accuracy: 66.4 at 2 rounds; 65.8 at 3 rounds; 64.2 at 4 rounds | Hurt |
| Replace LaViLa with LLaVA-NeXT as the captioner (ablation) | DrVideo · EgoSchema subset | accuracy: 62.6 to 61.2 | Hurt |
| Replace LaViLa with BLIP-2 as the captioner (ablation) | DrVideo · EgoSchema subset | accuracy: 62.6 to 59.6 | Hurt |
| Replace GPT-3.5 agents with Mistral-8x7B (ablation) | DrVideo · EgoSchema subset | accuracy: 62.6 to 47.6 | Hurt |
| Replace GPT-3.5 agents with GPT-4 (ablation) | DrVideo · EgoSchema subset | accuracy: 62.6 to 66.4 | Helped |
| Use DrVideo rather than VideoAgent | long videos in MovieChat-1K · MovieChat-1K global | accuracy: 65.4* to 93.1 | Helped |
| Use DrVideo rather than VideoAgent | long videos in MovieChat-1K · MovieChat-1K breakpoint | accuracy: 31.6* to 56.4 | Helped |
Surprising findings
- Using more initial key frames (K=10, 20) lowers accuracy, as added noisy information interferes with the LLM.
- Subtitles alone with DeepSeek yield 68.5% on Video-MME long split, close to the full DrVideo result with subtitles.
Limitations
- Performance declines beyond two agent interaction rounds due to accumulated noisy information.
- Depends heavily on the choice of LLM; open-source Mistral-8x7B performs much worse (47.6%) on EgoSchema.
- Relies on proprietary GPT models for agents, with API cost limiting ablations to GPT-3.5.
Experimental setup
Datasets: EgoSchema MovieChat-1K Video-MME
Benchmarks: EgoSchema MovieChat-1K Video-MME
Metrics: accuracy GPT-assisted accuracy GPT-assisted score
Baselines: LLoVi VideoAgent IG-VLM MoReVQA Vamos MovieChat MM-VID Claude 3.5 Sonnet GPT-4V GPT-4o mini Gemini 1.5 Flash VITA-8x7B
Builds on: LLoVi VideoAgent chain-of-thought prompting LLaVA-NeXT LaViLa
Closest papers by meaning
- Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding (NeurIPS 2025 · similarity 0.88)
- AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding (NeurIPS 2025 · similarity 0.88)
- Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension (NeurIPS 2025 · similarity 0.88)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.87)
- Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understanding (NeurIPS 2025 · similarity 0.87)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.