Research map / Video / Video-language understanding
Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vgent, a training-free graph-based RAG framework for long video understanding that builds an entity-linked clip graph offline and uses structured subquery verification before answer generation with open-source LVLMs.
Problem
Long video understanding with LVLMs is limited by context length and by RAG pipelines that break temporal dependencies across clips and retrieve noisy hard negatives.
Approach
Represent video clips as nodes in a graph linked by shared entities extracted by the LVLM, retrieve clips via query keywords, then verify each retrieved clip with structured yes/no or numerical subqueries and aggregate the verified information before final generation.
Key results
- On MLVU, Vgent improves LongVU-7B from 65.4 to 70.8 (+5.4) and Qwen2.5-VL-3B from 66.2 to 70.4 (+4.2).
- On Video-MME with subtitles, Qwen2.5-VL-7B + Vgent reaches 74.3 vs 71.1 base; the paper reports 8.6% over SoTA RAG methods on MLVU.
- Structured reasoning adds about 2.6% on MLVU and 1.6% on Video-MME over GraphRAG-only in the ablation for Qwen2.5-VL-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Vgent to Qwen2.5-VL 7B | Qwen2.5-VL · MLVU | accuracy: 68.8 -> 72.1 | Helped |
| Add Vgent to Qwen2.5-VL 7B | Qwen2.5-VL · VideoMME w/o subtitles | accuracy: 65.1 -> 68.9 | Helped |
| Add Vgent to Qwen2.5-VL 7B | Qwen2.5-VL · VideoMME w/ subtitles | accuracy: 71.1 -> 74.3 | Helped |
| Add Vgent to Qwen2.5-VL 7B | Qwen2.5-VL · LongVideoBench | accuracy: 56.0 -> 59.7 | Helped |
| Replace NaïveRAG with GraphRAG (ablation) | Qwen2.5-VL 7B · MLVU | accuracy: 65.4 -> 69.5 | Helped |
| Replace NaïveRAG with GraphRAG (ablation) | Qwen2.5-VL 7B · VideoMME | accuracy: 68.3 -> 72.7 | Helped |
| Add structured reasoning to GraphRAG (ablation) | Qwen2.5-VL 7B · MLVU | accuracy: 69.5 -> 72.1 | Helped |
| Add structured reasoning to GraphRAG (ablation) | Qwen2.5-VL 7B · VideoMME | accuracy: 72.7 -> 74.3 | Helped |
| Add structured reasoning to NaïveRAG (ablation) | Qwen2.5-VL 7B · MLVU | accuracy: 65.4 -> 68.6 | Helped |
| Add structured reasoning to NaïveRAG (ablation) | Qwen2.5-VL 7B · VideoMME | accuracy: 68.3 -> 69.8 | Helped |
| Use confidence-based refinement instead of structured reasoning (ablation) | Qwen2.5-VL + GraphRAG · MLVU | accuracy: 72.1 -> 69.5 | Hurt |
| Increase the refined clip cap from r=1 to r=5 (ablation) | Qwen2.5-VL + GraphRAG + SR · MLVU | accuracy: 63.2 -> 72.1 | Helped |
Surprising findings
- Confidence-based refinement (as in VideoAgent) gave only about 0.2% average improvement, so self-reported relevance was unreliable.
- Naive RAG alone reduced performance on MLVU (65.4 vs 68.8 base) and Video-MME (68.3 vs 71.1 base) for Qwen2.5-VL-7B.
Limitations
- Graph uses only textual entity descriptions, without visual embeddings or frame-level features.
- Performance is bounded by the capability of the underlying LVLM.
- Gains over base models on Video-MME are modest for some settings and no error bars or significance tests are reported.
Experimental setup
Datasets: MLVU Video-MME LongVideoBench
Benchmarks: MLVU Video-MME LongVideoBench
Metrics: accuracy
Baselines: NaïveRAG Video-RAG VideoAgent LLoVi DrVideo VideoTree GPT-4o Gemini 1.5 Pro
Builds on: retrieval-augmented generation VideoAgent Video-RAG GoldFish
Closest papers by meaning
- AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding (NeurIPS 2025 · similarity 0.93)
- Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension (NeurIPS 2025 · similarity 0.93)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.91)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.90)
- VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos (CVPR 2025 · similarity 0.90)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.